LAB · Evaluation
Evaluation Spec Builder
Design the dataset, segments, metrics, gates, and human review from the product risk.
- Suggested time
- 18 min
- Output artifact
- A JSON specification ready to become a regression suite.
- Privacy
- The definition is not sent to a server.
Local educational tool. Validate every result before production use.
When to use it
Eval designer
Use it before choosing metrics or graders, starting from failures that would harm the user or business.
How to use it well
- 1
Enter a real case without secrets or personal data.
- 2
Run, inspect, and challenge the assumptions; the output is not automatic truth.
- 3
Save the artifact in your project and turn it into a reviewable test or decision.
Related foundations
Harness Engineering
03 — Traces and the evaluation harness
Evaluating only the final text hides most important failures. An agent can write a correct answer after modifying the wrong resource, repeating a payment, or querying data it was not authorized to read.
Open foundation20 minLab 02 — Evaluation harness with isolated trials and separate graders
Localized walkthrough. Run the canonical public lab locally and preserve the result as evidence.
Open foundation7 minContinuous evaluation: eval pipelines and regression testing in CI
In classic software, if you don't touch the code, the behavior doesn't change. In LLM systems, there are four sources of change and you only control two:
Open foundation