AI engineering · Future AGI · 2026 · Python · uv · OpenTelemetry
AI Evaluation 1.0
One evaluate() call for every kind of check: local metrics in under a millisecond, cloud Turing models, or an LLM judge, routed automatically.
What I did
I rewrote the evaluation SDK for its 1.0 release: the unified evaluate() API with automatic engine routing, 72+ local metrics, and image and audio judging.
The local metrics cover string, JSON and similarity checks, NLI hallucination detection, RAG retrieval and generation, function calling, agent trajectories, structured output and guardrails. The release also added grading criteria generated from a short description, a feedback loop that turns corrections into few-shot examples, streaming evaluation with early stopping, OpenTelemetry spans, distributed backends (Celery, Ray, Temporal, Kubernetes), nine cookbooks, and the move from Poetry to uv.
Who I made it with
Evaluation is Future AGI team work; nik13 also worked on the release.
What it’s for
Teams check AI output in very different ways: a string match, a similarity score, a hallucination check, a judge model reading a screenshot. The SDK had grown a different API for each.
Version 1.0 puts them behind one call. evaluate() picks the engine (a local heuristic, a cloud model or an LLM-as-judge) and the same result shape comes back.
Why it was hardReplacing every earlier API with one function without breaking the people already calling them, while opening the same call to images and audio.
What changed for people using it
One import, from fi.evals import evaluate, instead of a different class per check. Text-only calls kept working, and passing an image or audio URL sends the same call to a multimodal judge.
Receipts
- future-agi/agent-learning-kit PR #13 — Version 1.0.0 (merged 27 Feb 2026)Unified evaluate() with engine routing, 72+ local metrics, multimodal judge, feedback loop, streaming, OTel spans, distributed backends, uv migration. Authored by Kartik; 38 of 39 commits his, one from nik13.
Next, a little further back in timeSimulate SDK →