One Capital Letter Was Silently Breaking My AI Support Bot, and It Wasn't in the New Model

The article details a real-world project using Weave to regression-test three OpenAI models on a banking support triage task. It illustrates how slight differences, such as a single capital letter in an intent label, can silently break AI-supported routing systems relying on exact JSON formats. The project uses 47 real customer queries from the BANKING77 dataset, applying two graders: a strict rule-based contract scorer checking exact JSON compliance, and a human-like AI judge scoring response quality. Three models—an older, a production, and a candidate—are compared, revealing that the production model repeatedly mislabels "request_refund" with a capital R, causing silent failures. The study highlights the necessity of both strict format checking and AI judgment in validating LLM outputs to avoid subtle bugs in deployed applications.