Evaluating AI Applications

"It seems better" is not a result. This course builds the measurement programme that turns a question about an AI application — did this change make it better or worse? — into evidence a colleague can check and a manager can act on. You start from a retrieval question-answering application you inherit, with two deliberately degraded twins kept beside it, because a metric that gives a good build and a broken build the same number is measuring nothing. You then build the programme piece by piece: a versioned golden dataset with provenance, tags, behaviour classes and a hash-based dev/held-out split that cannot be gamed; deterministic graders for citation, schema, abstention, refusal, pattern and length, with unit tests that fail in both directions; a rubric, a judge stand-in behind the provider interface, and the agreement arithmetic — percent agreement and Cohen's kappa — that tells you whether the rubric works at all; retrieval metrics (recall@k, precision@k, MRR) measured separately from answer quality, with sweeps over k and chunk size; a variant comparison with bootstrap confidence intervals and a per-case flip table; a fail-closed CI gate with per-tag floors and an explicit known-gap policy; a SQLite results database, run diffs and a triage workflow; token, cost and latency reporting driven from configuration rather than literals; and a safety scorecard built from an adversarial corpus. Everything runs offline against a local deterministic stub, with no API key anywhere and no network call to any model provider, which is what makes every number in the course reproducible to the digit. The course is explicit about what such a stub can and cannot tell you, and what would change with a real model. This is a learning pathway toward an AI application developer role. It does not promise employment, seniority, salary or any vendor certification; completing it earns an Ultiblob Certificate of Completion.