For traditional software, a regression is when a test that used to pass now fails. For AI features, a regression is when a model update or prompt change quietly makes the experience worse for a slice of users you have not tested. Without an eval suite, you will not know until churn tells you.
What an eval suite is
An eval suite is a structured set of representative inputs, expected behaviours, and a way to score the actual behaviour against them. It is not unit tests; it is closer to a regression test for a fuzzy system. It runs every time you change a prompt, model, retrieval pipeline, or routing rule, and it tells you whether you are improving or breaking things.
The discipline that makes them work
Evals are only as good as the inputs in them. They have to be drawn from real user interactions, including the edge cases that hurt. New failure modes from the wild get added back to the suite; passing tests stay there forever. The suite is a living product surface, not a one-off.
What to ship first
Start with twenty cases covering the most common user intents and the three or four edge cases that already burned you. That suite, run automatically on every change, prevents most of the regressions that quietly erode AI products. Expand from there.