After a prompt-template update, several previously passing test cases now produce unexpected outputs.

Regression tests determine whether behavior that previously met requirements still passes after a change. A stable reference set provides identical inputs, expected outcomes, and scoring rules across prompt versions, allowing the team to attribute failures to the update. Integration tests verify interactions among components but may not measure detailed output behavior. Adversarial tests target attacks and misuse cases, while smoke tests establish only that essential system paths remain available. The failed cases should be added to or retained within the regression suite, and the old and new prompt should be compared under the same model, parameters, tools, and retrieval context. Anthropic describes regression evaluations as protection against behavioral backsliding. Demystifying evals for AI agents