Evaluating it.

Anyone can show a RAG assistant answering one question. The hard part is knowing it's good, and proving it's safe, before real people rely on it. This is the eval layer under the search behind use case 01: what "good" had to mean, how I'd prove it never leaks, and the loop that catches a regression before a user does.

Before I'd trust it with real documents, I had to be able to say, with a number, that it was good, and separately that it was safe. Two different questions with two different bars: one you grade on a curve, one that can never slip.

I wrote down what "good" meant before I wrote the code, the way you write the test first. And I split the measures the way I learned to at Amazon: the inputs I can actually move, the outputs that tell me whether it's working, and the guardrails that can never slip. Keeping them in separate buckets matters, because you manage to inputs, you report on outputs, and you gate on guardrails. Treating any one like the others is how measurement goes wrong.

And every one gets a number. A metric without a target isn't a metric, it's a hope. Here's the bar I'd hold each to:

Good targets are one thing. An eval you can actually run, and trust, is another. A few habits separate an eval that helps from one that just makes you feel measured:

All of that is the eval you run before you ship: a fixed set of known cases, graded on every change, so a regression gets caught by me and not by an employee. But a gold set can't tell you what real people will actually ask, or how they'll react. Once it's live I watch what the gold set can't show: a thumbs up or down on each answer, and the honest one, whether the person still filed a ticket for the same thing after getting an answer. If they did, the answer didn't land, whatever it scored.

And none of it works unless every run leaves a trace: the question, a pseudonymous user id and the access groups it resolved to (never a name or email), which chunks got retrieved and how they scored, the model's output, and the citation. Enough to reproduce a failure, without logging personal data you'd then have to defend under GDPR, so it's minimized, access-controlled, and short-lived by design. Instrument that first, or you're debugging blind. Then read the traces. The numbers tell you something is off; only the trace tells you what. I'd sample real sessions every week, weighted toward the thumbs-down and the tickets raised anyway, group the failures into a handful of modes, fix the biggest bucket first, and turn each one into a new case in the gold set. That's the loop: production surfaces the failure, the failure becomes an offline test, and the test keeps it from coming back.

The way this usually starts is by hand: write sample questions and answers from the documents, then have people read the outputs and judge them. It works, and it doesn't scale. Most of it can be automated, but the judgment doesn't vanish. It moves.

So the human never actually leaves. They move from grading every answer to writing the rubric, calibrating the judge, reading the ambiguous failures, and owning the safety sign-off, because leak equals zero is a call I make, not one I hand to a model.

The word people worry about most is hallucination, when the model states something the policy doesn't say. I don't debug that in the abstract. I open the trace and ask one question: was the answer supported by what got retrieved? The answer sorts every failure into one of two kinds.

You almost never fix a hallucination in general. You decide which of the two it is, and fix that layer. Most of the prevention is already built in: answer only from the retrieved text, cite it, and abstain when nothing clears the bar. The eval is what tells you when that isn't holding.

The search eval graded one answer to one question. Once the search sat inside an assistant that handles parental leave and plans time off, that stopped being enough. The things that can go wrong now happen across turns and across people: a request drafted by one persona, approved by another, and reported back to the first. So the suite changed shape.

A case is now a sequence of steps, each as a named persona: a chat message, an API call, or a wait. Checks run on the reply's structure, not only its text. Did this message route to the agent or to search? Are there three date options and is the first one Dec 28? Is a draft pending, and is nothing submitted? Does the persona search exactly the documents they have access to, and is no other document named, flagged, or counted anywhere in the reply? The last one is the no-leak assertion, and it runs on every persona.

Forty-six cases across nine categories: routing, access, library, agent, leave, safety, flow, manager, and HR. State resets before each one. Cases that need the policy search running are skipped, not failed, when it is down, so a green run means what it says.

The suite earned its keep twice in its first day. The word "plan" in "executive equity plan" matched the time-off agent's intent pattern, so a policy question came back as three vacation options. And when the corpus grew to nine documents, the offline embedder started returning the India leave supplement for an executive-only question, above its relevance floor. The first was a one-line regex fix. The second became a grounding check: the top passage must share key terms with the question or the search abstains. Both showed up as red lines in a run, not as a bad moment in front of an audience.