Calibrating LLM judges against humans

An automated grader is only useful if it agrees with your experts. Here is how we measure that agreement. At Signa we spend most of our time reading traces, and most of what we know comes from failures our customers were brave enough to share.

Where it starts

Most teams begin with a spreadsheet of fifty questions and a thumbs-up column. It is a fine start and a poor finish: it measures the model, not the agent, and it never sees the long tail.

What we do instead

We score the whole run — retrieval, tool choice, arguments, final answer and policy — and we weight each case by how often it happens and how much it costs when it goes wrong.

Every automated judge is calibrated against human graders before its scores count toward a release gate.

What to take away

Measure the thing your customers experience, calibrate the thing that measures it, and keep the evidence. The rest is tooling.