For decades, software testing has relied on a comforting assumption: given the same input, systems should produce the same output. AI-enabled systems break that assumption entirely. Large language models (LLMs) and other AI components generate responses that can vary in structure, tone, and content while still appearing “correct.”

In this session, Lee Barnes will explore why traditional testing strategies struggle with non-deterministic AI behavior and where they quietly fail. Using real-world examples such as AI chatbots and resume-screening systems, he will walk attendees through practical techniques for validating AI outputs without relying on brittle, deterministic assertions. Topics include input variation strategies, semantic similarity analysis, bias detection, and using LLMs responsibly as automated evaluators, also known as “LLM-as-a-judge.”

Attendees will leave with a clear mental model for testing AI-based systems, concrete patterns they can apply immediately, and guidance on balancing automation, human judgment, and risk. If attendees are responsible for the quality of AI-driven features, this talk will help them move from uncertainty to confidence.