Why It Seemed Fine Is Not a Test
Trying a few examples by hand tells you almost nothing, for a reason that is easy to state and hard to feel. You choose the examples. You choose them from whatever you were already thinking about, you phrase them the way you would phrase them, and you judge the answers already knowing what you expected. Every part of that leans towards the thing working.
It also cannot detect the failure that matters most, which is getting worse. A change that improves most cases and breaks one you never checked looks like an improvement in the room and like a broken product to whoever hits the case that broke.
The fix is unfashionable and it works. Assemble a fixed set of real examples with known right answers, run it every time anything changes, and compare against the last run. That is the whole of evaluation. Most companies never do it, because it is dull and nobody outside the team ever asks for it.
Building the Set of Examples
Take them from real use rather than inventing them. Your support queue, your ticket history, your document archive, the questions people actually asked last month. Invented examples are too clean and too polite, and they are written by somebody who already knows what the system can do.
Cover the ordinary cases, then deliberately over include the awkward ones. Questions phrased badly. Questions built on a false premise, where the right answer contradicts the person asking. Questions where the correct response is that you cannot help. Questions with two reasonable interpretations. Those are where systems fail, and a set of clean questions will report success forever.
How many is enough is the wrong place to start. Start with what you can write correct answers for in a couple of days, and grow it from there. A modest set that genuinely represents your work beats a large one assembled by generating variations, because generated variations tend to share the blind spots of whatever generated them.
Write the expected answer and the conditions that make it right. Which document it should come from. What it must not say. Whether a refusal counts as acceptable. That detail is what lets a different person grade the run consistently a year from now.
Deciding What Counts as Right
Exact matching works for narrow tasks. Extracting an invoice number, choosing a category, pulling a date out of a sentence: one right answer, trivial grading. Anything producing prose needs a judgement about whether the answer is acceptable, and judgements need written rules or they drift.
Write the rules as things somebody can check rather than as qualities. Does it contain the key fact. Does it point at the right source. Does it avoid asserting anything the source does not say. Does it refuse where refusing was correct. A checklist of concrete conditions can be applied consistently by different people. A general impression of quality cannot.
Test your own grading before you trust it. Have two people grade the same run separately and compare the results. Where they disagree, your criteria are ambiguous, and ambiguous criteria produce numbers that move for reasons that have nothing to do with the system.
Using a model as the grader is practical and needs the same discipline. Check its grades against human grades on a sample before relying on it, and check it again now and then. A grader that has quietly drifted is worse than having no grader, because it reports confidently and nobody questions it.
Run It Before Every Change, Including the Safe Ones
The set earns its keep when it runs automatically on every change, as part of the normal pipeline. Anything that depends on somebody remembering will be skipped during the week when it matters most.
Changes that look safe are the ones that catch people out. Editing a prompt to fix a single case. Reorganising the documents. Moving to a newer model version because the old one is being retired. Adjusting how much material gets retrieved. Every one of those can shift behaviour across the whole system, and none of them look like a code change to anybody watching.
Compare against the previous run rather than against a target. What you want to see is which specific cases changed answer since last time. A total that stayed the same can hide a set of fixes and an equal set of breakages underneath it, and the breakages are what your customers will find first.
Keep the run fast enough that nobody resents it. An evaluation that takes hours gets moved to a nightly job, then to a weekly one, then to nobody. Where the full set is slow, split it. A quick subset on every change, the whole thing before a release, and the subset chosen to cover the parts that break most often.
Watching for the Day It Gets Worse
Things degrade for reasons that have nothing to do with your code. A provider updates a model behind the same name. Your own documents change, and a correction introduces an ambiguity. The questions people ask drift as your product moves. None of that produces an error in a log or a failed build.
So keep a small set of canary cases running on a schedule against production rather than only in the pipeline. When a case that has passed for months starts failing on a Tuesday and nobody deployed anything, you have learned something you could not have found any other way.
Keep the history. An evaluation set gets more valuable with every run behind it, because the useful question is nearly always whether this is worse than last month rather than whether it is good in the abstract. Store the results somewhere durable and boring, and keep them longer than you think you need to.
What to Measure in Production
The test set tells you about the cases you thought of. Production tells you about the ones you did not. You need both, and the production signals are usually available without building anything new.
Watch how often a conversation gets handed to a person, and more importantly the reason each time. Watch how often a customer asks the same thing twice, because a repeat is a failure nobody recorded. Watch reopened tickets. Where somebody edits a draft before sending it, capture the edit, because the difference between what was generated and what was actually sent is the most honest quality signal you will ever get.
Ratings from users are worth collecting and worth very little on their own. People rate rarely, and mostly they rate tone. Treat it as an early warning that something changed rather than as a measure of whether the thing is any good.
And read a sample by hand every week, permanently. Somebody senior enough to judge, a small number of real conversations, no dashboard involved. Every team that quietly stopped doing this found out later that something had been wrong for months.
Keep the Failures
Every failure is an addition to the test set. A customer complaint. A wrong answer somebody spotted. A case that had to be escalated because the system got confused. Capture the input and the correct answer, and add it permanently.
That habit is what stops the same mistake happening twice, and it separates a system that improves from one that only gets patched. Teams that do it end up with an evaluation set that is a genuine record of everything their business found difficult, and no vendor can hand you that.
Keep the fixed ones rather than removing them once they pass. Cases that were broken and repaired are the ones most likely to break again the next time somebody changes something nearby.
Record where each case came from as well. A failure a customer reported carries more weight than one an engineer imagined, and when you eventually have to choose which failures to fix first, knowing the source is what makes that choice defensible to whoever asks.
Somebody Has to Own It
Evaluation without an owner decays within a quarter. The set goes stale, runs get skipped when a release is urgent, and eventually somebody discovers the whole thing stopped running weeks ago and nobody noticed the missing report.
Name a person. Their job is to keep the set current, add new failures as they arrive, run the comparison before a release, and take the result to whoever decides whether to ship. The work is small. The ownership has to be real, with a name against it.
Give that person the authority to stop a release on the strength of the result. An evaluation anybody can overrule at the end of a quarter is a report rather than a control, and reports get filed.
Put the results somewhere the rest of the business can see them. One page carrying the last few runs does more for the credibility of this work than any amount of explaining, and it turns the day something gets worse into a shared fact rather than a conversation happening inside the engineering team.
