RAG Evaluation Harness: Build It, Explain It, Defend It · Edorax
← Catalog Instructor · Dev Weekends
RAG Evaluation Harness: Build It, Explain It, Defend It By Zeeshan Adil. Take apart one real, small evaluation system: an offline harness that compares prompt variants of a RAG support bot and recommends the safer one. Follow the evidence through one text normaliser and BM25, the negation-aware rule checks, a single blind LLM judge call with a schema, a replay cache and a labelled stub, and the safety-first decision with vetoes, tiers and ship blockers. Then see how validation, CI, task claims, storage and layered config prove a run, write your own third variant and help center, and practise defending every choice in interview form. Every exercise runs against the repository with no API key and no paid call. Build chapters need a laptop with Python 3.11; on a phone, each has a replay of a recorded run.
Difficulty Medium
Duration ~31 hours
Price Free
Explore the syllabus
Course curriculum 7 modules · 90 lessons. Open a module to see its lesson list.
01 Why offline evals matter 11 lessons View lessons Hide lessons 01.01 Set the scene: which prompt is safe to ship?15 min 01.01 Set the scene: which prompt is safe to ship?15 min 01.02 Why a RAG bot needs an offline gate25 min 01.02 Why a RAG bot needs an offline gate25 min 01.03 One run, five stages25 min 01.03 One run, five stages25 min What you'll build
A practice folder of scripts run against the harness, your own third variant and your own query run in stub mode with every pick explained, and a one-page interview cheat sheet you can say out loud.
01.04 Run the harness and read three outputs80 min
01.05 Interview warm-up: what it is and why30 min
01.06 Quiz: why offline evals10 min
01.07 Checklist and learning log25 min
01.07 Checklist and learning log25 min 02 Evidence: one normaliser, then BM25 10 lessons View lessons Hide lessons 02.01 Set the scene: a password document answers a screenshot question15 min 02.02 One normaliser for everything25 min 02.02 One normaliser for everything25 min 02.03 BM25 by hand20 min 02.03 BM25 by hand20 min 02.04 Score every document yourself60 min 02.05 Interview: why BM25 and not embeddings45 min 02.06 Quiz: retrieval and text10 min 02.07 Checklist and learning log25 min 02.07 Checklist and learning log25 min 03 Rule checks and the failure taxonomy 14 lessons View lessons Hide lessons 03.01 Set the scene: four rules read one answer15 min 03.02 Key facts and banned claims, with negation30 min 03.02 Key facts and banned claims, with negation30 min 03.03 The grounding score25 min 03.03 The grounding score25 min 03.04 Refusals, numbers, tone and relevance25 min 03.04 Refusals, numbers, tone and relevance25 min 03.05 The failure taxonomy20 min 03.05 The failure taxonomy20 min 03.06 Test the checks with your own cases75 min 03.07 Interview: why rules before a model45 min 03.08 Quiz: rule checks10 min 03.09 Checklist and learning log30 min 03.09 Checklist and learning log30 min 04 One LLM judge call 16 lessons View lessons Hide lessons 04.01 Set the scene: one call, five seconds, replayed since15 min 04.02 What the judge sees: blind labels and escaping25 min 04.02 What the judge sees: blind labels and escaping25 min 04.03 Judge bias, cost and calibration20 min 04.03 Judge bias, cost and calibration20 min 04.04 The reply schema, checked twice25 min 04.04 The reply schema, checked twice25 min 04.05 The request hash and replay20 min 04.05 The request hash and replay20 min 04.06 No key, bad replies and the stub25 min 04.06 No key, bad replies and the stub25 min 04.07 Run the judge without a key90 min 04.08 Interview: defending the judge45 min 04.09 Quiz: the judge10 min 04.10 Checklist and learning log30 min 04.10 Checklist and learning log30 min 05 The safety-first decision 12 lessons View lessons Hide lessons 05.01 Set the scene: what if the unsafe answer were clearer?15 min 05.02 Vetoes and tiers25 min 05.02 Vetoes and tiers25 min 05.03 The tie-break score and the ranking25 min 05.03 The tie-break score and the ranking25 min 05.04 Ship blockers, and clarity versus safety20 min 05.04 Ship blockers, and clarity versus safety20 min 05.05 Change the rules and watch the pick60 min 05.06 Interview: why safety beats clarity45 min 05.07 Quiz: the decision10 min 05.08 Checklist and learning log30 min 05.08 Checklist and learning log30 min 06 How the harness proves a run 14 lessons View lessons Hide lessons 06.01 Set the scene: CI fails if one output byte changes15 min 06.02 Validation, byte-stable files and CI25 min 06.02 Validation, byte-stable files and CI25 min 06.03 Pipeline stages that claim their task25 min 06.03 Pipeline stages that claim their task25 min 06.04 One table list, two databases20 min 06.04 One table list, two databases20 min 06.05 Five config layers20 min 06.05 Five config layers20 min 06.06 Tamper, claim, query and misconfigure90 min 06.07 Interview: engineering questions45 min 06.08 Quiz: proving a run10 min 06.09 Checklist and learning log30 min 06.09 Checklist and learning log30 min 07 Change it, and know its limits 13 lessons View lessons Hide lessons 07.01 Set the scene: every answer vetoed15 min 07.02 Inputs are data: the schema's rules25 min 07.02 Inputs are data: the schema's rules25 min 07.03 Adding a check: every place it touches20 min 07.03 Adding a check: every place it touches20 min 07.04 Write your own third variant60 min 07.05 Write a new help center90 min 07.06 Known limitations and what comes next25 min 07.06 Known limitations and what comes next25 min 07.07 Mock interview and your cheat sheet60 min 07.08 Quiz: the whole harness10 min 07.09 Checklist and learning log30 min 07.09 Checklist and learning log30 min