Loading course…
Instructor · Dev Weekends
By Zeeshan Adil. Learn the whole craft of evaluating an LLM product on one small app, the support assistant of an online grocery store: it classifies tickets, writes replies and runs a refund agent that calls tools. Read traces and name the failures, build a dataset that looks like production, write code graders, label replies and check an LLM judge against people, build your own harness with a replay cache, retries and resume, decide whether a difference is real, grade an agent with pass^k, wire a CI gate with cost and latency budgets, and monitor after launch. The capstone is a ship or no-ship review of a model upgrade, run with the graders, gate and runner you built. Every core exercise replays recorded model outputs, so no key and no paid call is needed. Build chapters need a laptop with Python 3.11 and a GitHub account; on a phone, each has a replay of a recorded run. RAG retrieval and grounding are taught in RAG Evaluation Harness, linked where the two meet.
Explore the syllabus
11 modules · 98 lessons. Open a module to see its lesson list.
What you'll build
Your own copy of the course repository with a failure taxonomy and criteria, a frozen dataset, audited code graders, a judge chosen on dev and reported on test, your own runner, an A against B verdict with intervals, agent graders with pass^k, a CI gate that blocks a planted regression, and a ship or no-ship report on a model upgrade that you can defend.
Have an account? Sign in to enroll