Know why your AI agents fail.
Verdict is observability for AI agents. It captures every run in full fidelity, lets you replay any run to prove a fix, and clusters failures automatically so you know what to repair first. Think OpenTelemetry, but for reasoning.
You shipped an agent. Now it fails silently.
A traditional web service fails loudly: a 500, a stack trace, a red line on a dashboard. An AI agent fails quietly. It returns a confident, fluent, wrong answer, and nothing about the response tells you which step went off the rails.
Silent failures. The agent gave a wrong or harmful answer. Nothing logged an error. You only find out when a user complains.
A record of every step the agent took, so you can see exactly where the reasoning turned.
No way to reproduce. The bad run is gone. You changed a prompt, but you can't confirm the fix without waiting for it to break again in production.
The ability to replay a real run against a change and see the new outcome, deterministically.
Cost and quality drift. Token spend creeps up. Answer quality slips. You learn about both from your users, not your dashboards.
Failures grouped by pattern and cost tracked per run, so drift shows up as a trend, not a surprise.
Standard APM tools were built for request/response services. They see an LLM call as one opaque HTTP request. They cannot see inside the reasoning: the retrieval that returned junk, the tool call that used a stale argument, the decision step that ignored its own context. That blind spot is the gap Verdict fills.
Three pillars, one job: make agent behavior legible.
Verdict instruments your agent, streams every step into a trace store, and gives you three things you can't get from logs alone.
Every step traced
Each run becomes a trajectory: an ordered timeline of steps (retrieval, LLM call, tool use, decision) with full inputs, outputs, tokens, and timing.
Replay any run
Take a real trajectory, change one step (swap the model, edit the prompt, override a decision) and re-execute from that point forward.
Failures clustered
A nightly job groups failing trajectories by similarity, so a hundred broken runs collapse into three root causes ranked by impact.
The data model, in one line
An Organization owns Projects. Each Project collects Trajectories (one per agent run). Each Trajectory is an ordered list of Steps. A run ends in one of three verdicts:
Verdicts: success, error, flagged. A run, step by step, with the failing step marked.
What you actually do with it.
- Debug a specific bad answer. Open the run's trajectory, walk the step timeline, find the retrieval step that pulled the wrong document. Root cause in minutes, not a guessing game.
- Prove a fix before you ship it. Replay ten failing runs with a new prompt and compare outcomes, without touching production.
- Triage a spike in failures. Open the clusters view: instead of reading 200 runs, you see three groups. Fix the biggest one first.
- Watch cost and quality trend. The dashboard tracks trajectory volume, failure rate, and estimated cost over 24h / 7d / 30d. Drift becomes a line, not a complaint.
- Find similar past incidents. Semantic search ranks trajectories by meaning, not keywords, so related runs surface even when the wording differs.
- Compare models or prompts. Replay the same real workload under a different model to see the quality and cost delta on your actual traffic.
How it's built.
Verdict is a small set of services, each with one job, connected by a trace store and an event stream. Your agent talks to one ingest endpoint; everything downstream is Verdict's problem.
- SDK: instruments your agent and ships each step to the ingest API. Minimal footprint on your code.
- verdict-ingest: the write path. Accepts steps, persists them to Postgres, drops events on an outbox for async processing so ingest stays fast and never blocks your agent.
- Workers: verdict-worker processes the outbox; verdict-replay-worker re-executes mutated steps; a nightly cron clusters failures; embeddings power semantic search.
- verdict-web: the Next.js App Router UI. Reads the trace store and subscribes to a server-sent-events stream so replays and new runs update live, no refresh.
- Logto SSO: authentication in front of the whole app; the API is reached through an authenticated proxy.
Design choice worth noting: ingest writes to an outbox and returns immediately. Heavy work (clustering, replay, embeddings) runs on separate workers. Tracing your agent adds negligible latency, and the expensive analysis never slows the write path or your production traffic.
From zero to a root cause.
Instrument your agent
Add the SDK and point it at your Verdict project. Each agent run now emits a trajectory of steps automatically.
Watch the dashboard
Trajectory volume, failure rate, estimated cost, and cluster count across 24h / 7d / 30d windows. Your at-a-glance health check.
Filter to what broke
Filter the trajectory list by status (error, flagged) or search semantically for the behavior you're chasing.
Open the trajectory
Walk the step timeline. Expand any step to see its full payload (prompt, response, retrieved docs, tokens).
Form a hypothesis, then replay
Mutate the suspect step: swap the model, edit the prompt, or override a decision. Re-execute it and everything after. Watch the new outcome stream in live.
Fix at the source, clear the cluster
Ship the change to your agent. The cluster it belonged to shrinks on the next nightly run. Drift you fixed stays fixed, and you can see it.
What it's worth.
Faster root cause
A step-level timeline turns 'the agent is wrong sometimes' into 'step 4 retrieved the wrong doc.' Debugging goes from hours to minutes.
Fixes you can prove
Replay validates a change against real failing runs before it ships. You stop shipping hopeful prompt edits and waiting to see if they helped.
Fix what matters first
Clustering ranks failures by how many runs they represent. Your effort goes to the root cause behind most failures, not the loudest ticket.
Cost visible early
Per-run token cost and trend charts mean spend creep is a line you watch, not a bill you're surprised by.
Agents are non-deterministic systems shipped into production with the debugging tools of deterministic ones. Verdict closes that gap. It doesn't make your agent smarter, it makes your agent legible, and legibility is the prerequisite for improving anything.
Beyond the engineer who built it.
- ML / platform teams get a shared source of truth for agent behavior across projects, instead of each engineer keeping their own log grep in their head.
- On-call & support can pull the exact trajectory behind a user complaint and hand engineering a reproducible case, not a vague report.
- Product & leadership get failure-rate and cost trends they can actually read: is quality improving release over release, and what is this agent costing us?
- Anyone evaluating a model change can replay real production traffic under the new model and see the delta on their own data, not a benchmark.
Ship agents you can actually debug.
Start tracing your agents in minutes. Free to start, no credit card.