Preprint · Future AGI · arXiv v1, 18 Sep 2025 · arXiv · cs.AI, cs.CL
AgentCompass
Once an agent is live, how do you find what went wrong across thousands of runs, and decide what to fix first?
What I did
First author, with Garvit Sapra, Rishav Hada and Nikhil Pareek at Future AGI.
Authors
NVJK Kartik, Garvit Sapra, Rishav Hada, Nikhil Pareek
Full title
AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production
A preprint: not yet peer-reviewed.
In plain words
Most agent evaluation happens before launch, on test sets. The expensive failures show up after launch: a tool quietly returns the wrong shape, the agent drifts off its goal, or it answers confidently from nothing.
AgentCompass reads a production trace the way an experienced debugger would. It finds the errors, groups them into themes, scores the run, and writes a summary that says what to fix first.
How it works
- A four-stage pipeline: error identification and categorisation → thematic clustering → quantitative scoring (e.g. factual grounding, safety, plan execution) → synthesis with a priority level.
- Errors are placed in a five-part taxonomy: Thinking & Response, Safety & Security, Tool & System, Workflow & Task Gaps, and Reflection Gaps.
- A dual memory: episodic (findings about one trace) and semantic (knowledge that generalises across traces, so recurring issues across a fleet of agents are recognised).
What it established
- On the public TRAIL benchmark, AgentCompass had the best error-localisation accuracy (GAIA split: 0.657 vs 0.546 for Gemini-2.5-Pro) and the best joint accuracy (0.239 vs 0.183).
- The paper also reports issues it found that the human annotations missed, and use with design partners on real deployments.
| Metric | AgentCompass | Gemini-2.5-Pro |
|---|---|---|
| GAIA · localisation acc. | ||
| GAIA · joint acc. | ||
| GAIA · category F1 · baseline better | ||
| GAIA · ρ with humans · baseline better | ||
| SWE · localisation acc. | ||
| SWE · joint acc. | ||
| SWE · category F1 | ||
| SWE · ρ with humans · baseline better |
Where it stops
- It is a preprint. It has not been peer-reviewed.
- On the GAIA split, Gemini-2.5-Pro and Gemini-2.5-Flash named the category better (category F1 0.389 and 0.337 vs 0.309), and five models tracked human scores more closely, Claude-3.7-Sonnet most (0.738 vs 0.430). On the SWE split, Gemini-2.5-Pro’s correlation with humans was twice as high (0.817 vs 0.408), and joint accuracy was a tie (0.051 vs 0.050).
- The other models’ numbers are the ones reported in the TRAIL paper; they weren’t re-run for this comparison.
- The design-partner results are described qualitatively, without public numbers.
Receipts
- arXiv:2509.14647 — AgentCompass (preprint, v1, 18 Sep 2025)Title, author order (Kartik first), date, preprint status, abstract.
- AgentCompass full text (arXiv HTML)Four-stage pipeline, five-category error taxonomy, episodic/semantic memory, TRAIL results table.
Next, a little further back in timeThe open-source SDKs →