N.V.J.K KartikKartik

Platform engineering · Future AGI · 2026 · Django · ClickHouse · Postgres

Annotations on ClickHouse

Moving the annotation system’s reads onto ClickHouse ahead of the Postgres tracer tables being dropped, and keeping it standing: 20 merged PRs over July 2026.

Plate: reads moving from Postgres to ClickHouse.

What I did

I was responsible for the annotation side of the migration. Over four weeks that was 20 merged PRs, 16 on annotations and 4 on Error Feed and its scanner: every annotation read path moved to ClickHouse (resolution, previews, availability checks, scores, export, and the filter-mode adds), Error Feed moved too, memory guardrails so ClickHouse reads stopped running out of memory on large voice spans and in the trace scanner, project scoping on every queue read, and a performance sweep of the queue screens that removed N+1 queries, per-row reads and per-item writes.

The core PR moved the subsystem’s reads of trace, span and session data to ClickHouse. Reads are tenant-gated and fail closed.

Trace roots are read in lean batches so large queues don’t run out of memory. The PR replaced my own earlier draft, which only guarded the Postgres reads, since removing them was the right end state once the tables were gone.

The core PR, #1495 ↗All my merged PRs ↗

Who I made it with

One part of the team’s wider migration of tracing to ClickHouse.

The call I made

Remove the old path, don’t guard it

The Postgres tracer tables were being dropped. My first draft guarded every read against the table going missing; the right end state was to stop reading them at all, and to fail closed rather than open.

Before
  1. Annotation readresolve, preview, score
  2. Postgres tracer tablesbeing dropped
  3. Catch the missing tablea savepoint and a 42P01 catch
  4. A guard around a table that won’t exist
After
  1. Annotation readresolve, preview, score
  2. Straight to ClickHouselean, batched reads
  3. Checked against the tenantfails closed
  4. Large queues stay in memorytrace roots read in batches

What it’s for

Tracing at Future AGI was moving fully to ClickHouse, and the Postgres trace, span and session tables were going away. The annotation system (queues, scores, span notes) still read trace data from Postgres.

Why it was hard

Removing a data source from under a live feature. Every read path had to move without leaking data between tenants or failing open.

In numbers

20
merged PRs, 3–31 July 2026: reads, memory, scoping and speed. Source16 on annotations; 4 on Error Feed and its trace scanner, which moved in the same migration.
575
annotation tests passing across 12 suites in the core PR. Source

Receipts

  1. future-agi/future-agi PR #1495 — ClickHouse-native reads for annotation sources (merged 11 Jul 2026)Annotation reads of trace, span and session data moved to ClickHouse; tenant-gated, fail-closed; lean batched reads; supersedes the drop-safety draft #1214, which guarded the Postgres reads instead of removing them. 575 annotation tests passing across 12 suites. Authored by Kartik.
  2. future-agi/future-agi — Kartik’s merged annotation and ClickHouse PRs20 merged PRs, 3–31 Jul 2026: 16 on annotations, 4 on Error Feed and its scanner. Annotation reads to ClickHouse: #1427, #1495, #1604, #1607. Error Feed: #1510, #1644. Memory: #1160, #1434 (annotations), #1373, #1455 (trace scanner). Project scoping: #1565. Queue performance: #1591, #1593, #1831, #1852, #1861, #1865, #1871, #1876, #1878. The link lists all of Kartik’s merged PRs in the repository, not only these 20.

Every source on the site

Next, a little further back in timeError Feed →