Evaluation Harness

An evaluation harness is the test framework used to measure an AI system against a fixed set of inputs, expected outputs, and metrics, run on every change.

What is Evaluation Harness?

An evaluation harness combines a curated dataset, scoring functions, and a runner. Scoring may be exact-match, embedding similarity, rubric-based LLM-as-judge, business KPI, or human review. The harness runs on every prompt change, model change, retrieval change, or data change. Without it, the team has no way to tell whether an edit improved or regressed the system.

How does Evaluation Harness apply to enterprise AI?

Enterprise AI systems must have an evaluation harness before they go live. It is the difference between a demo and a production system, and the artefact regulators ask for under the EU AI Act conformity assessment.