RAG Evaluation Harness: Build It, Explain It, Defend It
Written by Zeeshan Adil
8 h 40 min · 7 parts · 52 chapters
From chapter 1, 8 h 40 min
By Zeeshan Adil. Take apart one real, small evaluation system: an offline harness that compares prompt variants of a RAG support bot and recommends the safer one. Follow the evidence through one text normaliser and BM25, the negation-aware rule checks, a single blind LLM judge call with a schema, a replay cache and a labelled stub, and the safety-first decision with vetoes, tiers and ship blockers. Then see how validation, CI, task claims, storage and layered config prove a run, write your own third variant and help center, and practise defending every choice in interview form. Every exercise runs against the repository with no API key and no paid call. Build chapters need a laptop with Python 3.11; on a phone, each has a replay of a recorded run.
Read the course: the same chapters with diagrams, labs and quizzes.