AI promises magical testing gains during demos and proofs of concept (PoCs), but the value often disappears behind vanity metrics and vendor sparkle. Generative and agentic tools promise “10x testing,” but do they really move the business needle?
Borrowing the “Now You See Me, Now You Don’t” metaphor, Usha Kandala will delve into the world of “magical” key performance indicators (KPIs) and metrics in AI, helping the audience pull back the curtain and turn flashy illusions into measurable, auditable impact.

Attendees will learn how to separate the magic trick from the method by showing how to measure real return on investment (ROI) across two lenses:

AI-assisted testing, where testers use AI to think, design, and review faster, but the human in the loop still makes the final call. The value shows up as time saved per tester, faster reviews, and more high-severity bugs found with less rework. Usha will show how to measure this with simple, auditable signals like severity-weighted defects per hour and acceptance vs. override of AI suggestions, keeping hallucinations in check so speed never compromises quality.

AI testing tools, where autonomous/agentic AI creates and runs tests on its own in continuous integration/continuous delivery (CI/CD), aiming for wider, more reliable coverage and earlier defect signals. She will show how to prove impact by tracking useful flow coverage, stability after removing flakes, and the time from code change to first real failure, plus how precise those failures are, ensuring the ROI comes from true signal, not noisy alerts.

By the end of this session, attendees will learn how to replace stage tricks with a baseline method based on what they already have and tell a CFO-ready story that earns budget.

The audience will learn how to run a 30-day lighthouse experiment with a control group, build a one-page KPI map to catch misdirection, and use a scorecard that makes “now you see me?” become “now you can’t miss me.”