AI Evals and Harnesses: Measure It, Trust It, Ship It
Written by Zeeshan Adil
10 h 48 min · 11 parts · 76 chapters
From chapter 1, 10 h 48 min
By Zeeshan Adil. Learn the whole craft of evaluating an LLM product on one small app, the support assistant of an online grocery store: it classifies tickets, writes replies and runs a refund agent that calls tools. Read traces and name the failures, build a dataset that looks like production, write code graders, label replies and check an LLM judge against people, build your own harness with a replay cache, retries and resume, decide whether a difference is real, grade an agent with pass^k, wire a CI gate with cost and latency budgets, and monitor after launch. The capstone is a ship or no-ship review of a model upgrade, run with the graders, gate and runner you built. Every core exercise replays recorded model outputs, so no key and no paid call is needed. Build chapters need a laptop with Python 3.11 and a GitHub account; on a phone, each has a replay of a recorded run. RAG retrieval and grounding are taught in RAG Evaluation Harness, linked where the two meet.
Read the course: the same chapters with diagrams, labs and quizzes.
Chapters
Part 1 · Why evals, and what a harness does
43 min
Part 2 · Look at the data first
47 min
Part 3 · A dataset that looks like production
48 min
Part 4 · Code graders and the numbers they feed
47 min
Part 5 · Labels and LLM judges, checked against people