00 // Agent reliability for regulated AI
Evals, tracing and guardrails for teams shipping AI agents where mistakes are regulated — banking, insurance and health.
NORTHVALE
Corvid Health
LEDGERLINE
Northvale Bank
OSTRA
Quayside
Albion Mutual
Albion Mutual
02 // Why we exist
Most AI failures are not crashes. They are fluent, confident answers that break a policy nobody tested. We make them visible before they ship.
We started Signa after watching a chatbot approve a loan it should have declined. Today 38 regulated teams run every agent change through us — from the first prompt to the last audit request.
— Ines Marlow, Co-founder & CEO
02 / Teams in production
38
From one support bot to fleets of tool-using agents.
03 / Failures caught pre-release
99.3
%
Rolling 90-day average, audited.
04 / Median rollout
14
days
Kick-off → production sign-off.
05 // Platform
Four layers between your agent and an incident.
Evals, traces and guardrails share one data model — so every production failure becomes tomorrow’s test case.
01
Observe
Tracing
Every agent run captured as a replayable trace — prompts, tool calls, retrievals and outputs, linked to the user and the model version.
100%
of agent runs captured and replayable
→
02
Measure
Evals
Regression suites that score every prompt, model and tool change against your own graded cases — before a single customer sees it.
3,400
graded cases in a typical bank suite
→
03
Protect
Guardrails
Inline policy checks on every response — PII, jailbreaks, off-policy advice and hallucinated facts — in under 40 ms at the p95.
38 ms
p95 added latency per checked response
→
04
Prove
Audit
Evidence packs mapped to model-risk policy and the EU AI Act — every test, threshold and incident, exportable on demand.
2 days
typical model-risk review, down from 3 weeks
→
06 // From pilot to production
Six weeks. One audit trail.
Instrumentation, baselines, red-teaming and guardrails run in parallel on one plan — that is how teams reach production in weeks, not quarters.
W1
W2
W3
W4
W5
W6
W7
W8
Connect & trace
W01
Guardrails live
W01–W03
Red-teaming
W02–W04
Baseline evals
W03–W05
Shadow traffic
W04–W06
Sign-off → production
W06
07 // Featured customer
Northvale Mortgage Assistant
A retail bank moved its mortgage assistant from pilot to 1.2M customers with a full audit trail in six weeks.
Reach
1.2M customers
Region
London, UK
In production
Q2 2026
Pass rate
99.2%
08 // Customers
Risk teams sleep better with Signa in the loop.
“Signa caught a refund-policy hallucination in shadow traffic two days before launch. Our regulator asked how we knew. We showed them the trace.”
PR
Priya Raman
VP AI Platform, Northvale Bank
“One trace from prompt to payout. Our model-risk review went from three weeks to two days.”
DO
Dana Okafor
Head of Model Risk, Albion Mutual
Traces evaluated
4.2 bn
evaluated for banks, insurers and health providers — with zero data leaving their cloud.
10 // Journal
Notes from the eval lab.
Methods
Why accuracy is the wrong metric for agents
Accuracy on a static set tells you little about an agent that calls tools. Here is what we measure instead — and why.
Regulation
What the EU AI Act actually asks of your chatbot
Fewer obligations than the headlines suggest, more evidence than most teams keep. A practical reading.
Post-mortem
Anatomy of a hallucinated refund
One trace, eleven spans and a retrieval step that returned the wrong policy. How we found it in four minutes.
Methods
Calibrating LLM judges against humans
An automated grader is only useful if it agrees with your experts. Here is how we measure that agreement.
Engineering
Guardrails in under 40 milliseconds
Policy checks that add no noticeable latency: small models, early exits and a lot of caching.
Guides
Shadow traffic: a practical guide
Mirror real requests to a new agent, score everything, change nothing. The safest way to find out what a release will do.
11 // Start here
Ship the agent. Keep the evidence.
Send us an agent, a failure or a policy. An evaluation engineer replies within one working day.