N.V.J.K KartikKartik

AI engineering · Future AGI · 2026 — now · Python · Django · ClickHouse · React

Error Feed

A harness that lives alongside the product: it watches the traces, groups failures (failed evals included) into issues, investigates each one with an agent, and shows the causal trail.

Plate: failing traces with one cluster circled, and the cause, fix, confidence and evidence the agent returns.

What I did

I built the investigation agent: it reads a cluster’s traces from ClickHouse, compares them across version, model and region, and returns a two-sentence cause, a one-sentence fix, a confidence level and the evidence. The result is cached and streamed into a Fix tab.

I kept it alive through the migration: every feed read, the deep-analysis worker and the live scanner moved to ClickHouse, so the feed survived the Postgres tracer tables being dropped. I proved it by dropping the tables in a test and renaming them away on a live stack.

I made it fast at scale: on a large tenant the feed took 15–27 seconds and the traces tab never returned. Pruning the reads by index and project brought every endpoint to about 150–200 ms.

Then I took the model off the agent’s read path. Timed on a live run, ClickHouse was 551 ms of a five-minute investigation and model turns were 88% of it, so the agent now reads each trace’s span tree directly, within a budget per trace, instead of waiting for a cheaper model to paraphrase it.

In September I handed trace investigation to a separate worker with durable jobs and stored evidence, turned grouping on by default for eligible projects, extended clustering to choice, threshold and numeric eval scores, and gave each finding its causal trail: where it started, the decisive step, and the symptom. It shipped in v1.40.0 behind a kill-switch and budget limits.

Where it started: AgentCompassThe agent, #853 ↗ClickHouse-native, #1510 ↗Fast at scale, #1644 ↗Read path, #2019 ↗Investigation handoff, #2942 ↗Grouping and breadcrumbs, #2979 ↗

Who I made it with

Error Feed is a Future AGI team product. velalagan-pixel built the redesigned Overview and Fix tab interface for the investigation, hadarishav worked on the scanner and clustering, and atharva-bhange built the grouping and severity jobs in the September release.

The call I made

Take the model off the read path

The agent first asked a cheaper model to summarise each trace before reasoning over it. Timed on a live run, ClickHouse was 551 ms of a five-minute investigation and model turns were 88%. The cost worth removing was the round trip, not the query, so the read became deterministic.

Before
  1. The agent reads a trace
  2. Whole span tree sent to a cheaper modelno cap on spans or size
  3. A skeleton comes backwithout the inputs and outputs
  4. Model turns are 88% of the run
After
  1. The agent reads a tracefrom ClickHouse
  2. Span tree rendered directlyno model on the read path
  3. One budget per tracefailing spans claim it first
  4. The main model reasons over real inputs and outputs

What it’s for

Every agent run leaves a trace. Error Feed reads them continuously, groups the failures into issues, and for each one says what broke, why, and what to fix.

Before it, someone had to open traces one by one, compare them and spot the pattern themselves.

It grew out of AgentCompass, the trace evaluator I wrote the paper on.

Why it was hard

Finding what is common to every failing trace, not just the first few, without reading thousands of them one by one, on tables with millions of spans.

In numbers

17 s → 0.2 s
cluster detail, one of five feed endpoints brought to ~150–200 ms. SourceBenchmarked on a real 10M-row spans table with 120 seeded clusters, warm: list 19.7 s → ~150 ms, detail 17 s → ~200 ms, overview 15–17 s → ~200 ms, trends 14–24 s → ~160 ms, traces tab >30 s → ~150 ms.
65–92%
lower feed API latencies after the first performance pass. SourceFour feed endpoints, before and after the PR: overview ~1.5 s → 49 ms, list ~1.7 s → 47 ms, sidebar ~1.2 s → 100 ms, trends ~800 ms → 280 ms. The PR doesn’t state the load they were measured under.
$0.03–0.08
per investigation at launch (June 2026), the PR’s estimate. SourceEstimated for the launch design, when a lite model summarised each trace first. The agent has changed since, and there’s no newer public figure.
2,077
GitHub stars on the open-source platform it ships in (Sep 2026). SourceThe platform’s reach, not a measure of my part in it.

Receipts

  1. future-agi/future-agi PR #853 — Error Feed cluster RCA agent, perf sweep, billing (merged 27 Jun 2026)Root-cause agent over ClickHouse traces (at launch: per-trace summaries on a lite model, the investigation on the main model, ~$0.03–0.08 a run), streamed Fix tab, cached synthesis, billing wiring. Feed endpoints before → after: overview ~1.5 s → 49 ms, trends ~800 ms → 280 ms, sidebar ~1.2 s → 100 ms, list ~1.7 s → 47 ms. Authored by Kartik: 83 of 113 commits. velalagan-pixel (19) built the redesigned Overview and Fix tab UI; the KarthikAvinashFI, commitPirate and cdileep23 commits are unrelated dev-branch fixes carried in by a merge. The agent itself is in ee (internal ee#121, all 54 commits by Kartik). It also folds Gemini thinking tokens into completion tokens, which the cost header had left out.
  2. future-agi/future-agi PR #1510 — make Error Feed ClickHouse-native (merged 11 Jul 2026)Every feed read, the deep-analysis worker and the live scanner moved to ClickHouse to survive the tracer Postgres-table drop; proven by dropping the tables in tests and renaming them away on a live stack. Authored by Kartik.
  3. future-agi/future-agi PR #1644 — prune feed ClickHouse reads (merged 18 Jul 2026)Benchmarked on a 10M-row spans table: list 19.7 s → ~150 ms, cluster detail 17 s → ~200 ms, overview 15–17 s → ~200 ms, trends 14–24 s → ~160 ms, traces tab >30 s → ~150 ms. Authored by Kartik.
  4. future-agi/future-agi PR #2019 — scanner and cluster-RCA port, deterministic RCA read path (merged 7 Aug 2026)Measured on a live production run: ClickHouse was 551 ms of a five-minute investigation and 88% of wall clock was model turns. The read tool stopped sending the span tree to a cheaper model; it renders budgeted verbatim I/O deterministically, budgeted per trace with failing spans first, plus a 90 s per-turn deadline. Authored by Kartik (21 of 24 commits; 3 ported from hadarishav’s scanner branch).
  5. future-agi/future-agi PR #2942 — Error Feed investigation handoff (merged 23 Sep 2026, released in v1.40.0)Opt-in traces are handed to a separate investigation worker; durable investigation jobs, normalized findings and evidence are stored and read back in Error Feed and cluster RCA; legacy scans are backfilled. Authored by Kartik (32 of 32 commits). In the v1.40.0 release (23 Sep 2026), 47 of 59 commits are Kartik’s and 12 are atharva-bhange’s (grouping jobs, severity and fix-layer assessments).
  6. future-agi/future-agi PR #2979 — grouping, scored evals and causal breadcrumbs (merged 23 Sep 2026)Grouping on by default for eligible projects, with a kill-switch and budget controls; choice, threshold and numeric eval failures eligible for clustering; each finding shows its origin, decisive and symptom steps. Authored by Kartik (21 of 23 commits; atharva-bhange 2). Merged into a stacked branch and released in v1.40.0 (23 Sep 2026); the PR notes quality at scale was still to be verified after deployment.
  7. GitHub — future-agi/future-agiThe open-source platform repository Error Feed and the annotation reads ship in: 2,077 stars and 644 forks on 26 Sep 2026.

Every source on the site

Next, a little further back in timePosts about what broke →