Anatomy of a hallucinated refund

One trace, eleven spans and a retrieval step that returned the wrong policy. How we found it in four minutes. At Signa we spend most of our time reading traces, and most of what we know comes from failures our customers were brave enough to share.

Where it starts

Most teams begin with a spreadsheet of fifty questions and a thumbs-up column. It is a fine start and a poor finish: it measures the model, not the agent, and it never sees the long tail.

What we do instead

We score the whole run — retrieval, tool choice, arguments, final answer and policy — and we weight each case by how often it happens and how much it costs when it goes wrong.

Every automated judge is calibrated against human graders before its scores count toward a release gate.

What to take away

Measure the thing your customers experience, calibrate the thing that measures it, and keep the evidence. The rest is tooling.