Research system · Future AGI · 2025 · arXiv 2509.14647 · preprint
AgentCompass
An evaluator that reads production agent traces the way an experienced debugger would: find the error, name it, cluster it, score it, say what to fix first.
What I did
First author of the paper, with Garvit Sapra, Rishav Hada and Nikhil Pareek.
I also built the production agent the paper describes, in September 2025: four passes over a trace, subtrees for traces past the token limit, episodic and semantic memory, and HDBSCAN clustering. Error Feed grew out of it.
The “Break the agent” demo on this page is my own illustration of the idea, built from a real bug I wrote about. It isn’t the product and it isn’t benchmark data.
Who I made it with
Garvit Sapra, Rishav Hada and Nikhil Pareek (Future AGI).
What it’s for
Pre-launch evals test the cases you thought of. Production is full of the ones you didn’t. AgentCompass is built for after deployment, when the question stops being “does it pass?” and becomes “what is it doing wrong, and how often?”
Why it was hardProduction failures are rarely one bug. They are patterns spread across thousands of runs, and the evaluator has to remember what it has already seen.
Memory
Episodic memory keeps what was found in a single trace. Semantic memory keeps what generalises, so the tenth occurrence of a failure across a fleet of agents is recognised as a pattern, not a new mystery.
Where it fell short
On TRAIL it was best at locating errors. On the GAIA split two Gemini models named the category better, and five models tracked human scores more closely, Claude-3.7-Sonnet most (0.738 vs 0.430). On the SWE split Gemini-2.5-Pro’s correlation with humans was twice as high (0.817 vs 0.408). The paper is a preprint. The research page has the full table.
In numbers
More from the project
Try it: break the agent
Pick a scenario, click any span to inspect it, then run the evaluation. Watch what the dashboard says while the answer is wrong.
Illustrative demo A fictional support agent, deterministic and local. It’s modelled on a real bug I wrote about. Not product output, not benchmark data.
Retrieval finds three help articles. The model answers from them.
- HTTP 200
- Latency 1.84 s
- Error rate 0.0%
- Grounded? not measured
user How do I reset two-factor auth?
Open Settings → Sign-in & security → Two-factor authentication and choose Reset. You’ll confirm with your password and a code sent to your recovery email.
- 01 Identify
- 02 Localise
- 03 Cluster
- 04 Score & summarise
Receipts
- arXiv:2509.14647 — AgentCompass (preprint, v1, 18 Sep 2025)Title, author order (Kartik first), date, preprint status, abstract.
- AgentCompass full text (arXiv HTML)Four-stage pipeline, five-category error taxonomy, episodic/semantic memory, TRAIL results table.
- Future AGI: trace error-analysis agent (futureagi/ee/agenthub/traceerroragent)The public, EE-licensed code, which Error Feed’s error analysis and investigation agent build on. Authorship is from internal PRs agentic-eval#338 and core-backend#1328 (merged 4 Sep 2025; 14 of 16 and 19 of 21 commits by Kartik, the rest gsapra), which a visitor can’t open: four passes over a trace, subtree batching over the token limit, episodic and semantic memory, HDBSCAN clustering.
Next, a little further back in timeThe open-source SDKs →