Open curriculum · verified editionContribute on GitHub
LLM EngineerClub
SearchESClub Quiz
Lesson7 min1,378 words

05 — Prompt Evaluation: Test Datasets, LLM-as-a-Judge, and A/B Testing

Associated Labs: ../labs/06_eval_prompts_dataset.py and ../labs/07_ab_testing_prompts.py This is the core topic of the module: your milestone is a functioning evaluation pipeline.

SourceImprove this page

Associated Labs: ../labs/06_eval_prompts_dataset.py and ../labs/07_ab_testing_prompts.py This is the core topic of the module: your milestone is a functioning evaluation pipeline.

The "It Works for Me" Problem#

The default workflow for prompts is: test 3 inputs manually, the output "looks good," and ship to production. This has two structural flaws:

  1. Tiny and biased sample: your 3 test inputs do not represent the real distribution (no one manually tests tickets written in all caps, with two mixed topics, or in Catalan).
  2. No memory: when you tweak the prompt next month, you won't know if you improved the new case and broke five old ones.

The solution is the same one software solved decades ago: tests. A prompt is code; it is evaluated with datasets, metrics, and regression. The difference with classic software is that the output is stochastic and sometimes there is no single "correct answer" — hence the three tools in this topic.

1. Test Datasets (Golden Datasets)#

An evaluation dataset is a list of cases {input, resultado_esperado} (plus metadata). For the ticket classification in the module:

{
  "id": "t-014",
  "texto": "ME HABÉIS COBRADO DOS VECES!!! quiero mi dinero YA",
  "categoria_esperada": "facturacion",
  "dificultad": "facil",
  "nota": "tono agresivo, no debe confundir con 'cuenta'"
}

How to build it:

  • Start with 20–50 cases. This is sufficient to detect large differences between prompts and fits anyone's budget. Grow to 200+ over time.
  • Sources in order of quality: (1) anonymized real production data, (2) reported failure cases, (3) synthetic cases generated by an LLM and manually reviewed — review is not optional: the generator makes the same systematic errors you will later want to detect.
  • Cover the ugly distribution: edge cases between classes, ambiguous inputs, poorly written text, empty or out-of-domain inputs, injection attempts. A dataset of pretty cases measures little.
  • Label with documented criteria: if two people would label a case differently, write the tie-breaking rule in the dataset itself (field nota). This criteria document is gold: it is what you will later give to the LLM-judge.
  • Freeze and version it alongside the prompt (topic 06). If you change cases, change the version: comparing accuracy across different datasets means nothing.

Programmatic metrics (when there is a correct answer):

Task Metric
Classification accuracy, and per-class precision/recall if there is imbalance
Extraction exact match per field, F1 over fields
Structured output % of valid parses + per-field metrics
Generation with constraints programmatic checks: length, presence of sections, regex

Always run with temperature=0 (or the minimum) to evaluate: you want to measure the prompt, not sampling noise. If the production task uses high temperature, evaluate with N samples per case and average.

2. LLM-as-judge#

For outputs without a single correct answer (summaries, customer responses, explanations), programmatic metrics fall short. Classic NLP metrics (BLEU, ROUGE) correlate poorly with actual quality. The current standard practice: use an LLM as an evaluator, formalized in Zheng et al., 2023 (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena), which found that strong LLM judges achieve >80% agreement with human preferences — comparable to human-to-human agreement.

Three modes of judgment:

  1. Rubric-based scoring (single answer grading): "score from 1 to 5 based on these criteria". Useful for monitoring over time.
  2. Pairwise comparison (pairwise): "response A or response B?" Much more reliable than absolute scoring (LLMs score with inconsistent scales, but compare well). This is the basis for the A/B testing below.
  3. Binary criterion verification: "does the response mention the 14-day deadline? Does it contain any refund promise?" — the most robust of the three modes, because each check is almost programmatic.

Writing a good judge prompt#

The judge is a prompt with its own rules:

Eres un evaluador de respuestas de soporte al cliente de un SaaS de
facturación. Evalúa la respuesta según esta rúbrica, criterio a criterio:

<rubrica>
1. Corrección: ¿es exacta según la política de la empresa? (sin inventar
   funcionalidades ni prometer reembolsos)
2. Completitud: ¿responde a todo lo que preguntó el cliente?
3. Tono: ¿profesional, empático, en español correcto?
</rubrica>

<pregunta_cliente>{{question}}</pregunta_cliente>
<respuesta_a_evaluar>{{answer}}</respuesta_a_evaluar>

Analiza cada criterio en <analisis>, después da un veredicto JSON:
{"correccion": 1-5, "completitud": 1-5, "tono": 1-5, "veredicto_global": "aprobado|rechazado"}

Judge rules:

  • Explicit and decomposed rubric. "Evaluate the quality" produces noise; concrete criteria produce signal. Leverage the dataset's criterion document.
  • Reasoning before the verdict (Judge CoT): improves reliability and allows you to audit the judgments.
  • Judge ≥ Evaluated: use a model equal to or more capable than the one generating as the judge. Judging is easier than generating, but a weak judge adds its own noise.
  • Structured output from the judge (topic 04): the verdict must be parseable for aggregation.

Known judge biases (and mitigations)#

Documented in the Zheng et al. paper itself:

Bias What happens Mitigation
Position In pairwise comparisons, systematically favors the first (or second) option Evaluate each pair twice with inverted order; if verdicts differ, count as a tie
Verbosity Prefers longer responses for identical content Rubric with explicit conciseness criterion; compare lengths in the analysis
Self-preference A model tends to prefer outputs from its own family Judge from a different family than the generator, or two judges
Anchoring on form Beautiful formatting (markdown, lists) inflates the score Ask the judge to ignore format if it is not a criterion

And the meta-rule: calibrate the judge against humans once. Manually label 20–30 cases, run the judge, measure agreement. If the judge does not agree with you in ≥80–90%, fix the judge's prompt before trusting its numbers.

3. Prompt A/B testing#

With a dataset + metric (programmatic or judge), comparing prompt A with B is an experiment:

para cada caso del dataset:
    salida_A = llm(prompt_A, caso)
    salida_B = llm(prompt_B, caso)
    veredicto = comparar(salida_A, salida_B)   # métrica o juez pairwise (×2 órdenes)

agregar: % victorias A, % victorias B, % empates

The detail that almost everyone skips: is the difference real or noise? With 25 cases, where A wins 14–11 means nothing. Minimum criterion without needing a statistics course:

  • Use a sign test or binomial test on the non-tie cases: if A wins w out of n decided comparisons, under the null hypothesis (prompts are equal) w follows a Binomial(n, 0.5). With n=20 decided, you need ~15 wins (75%) for p<0.05.
  • Practical translation: with 20–50 case datasets you only detect large differences. For fine differences, expand the dataset before drawing conclusions.
  • Always report the three numbers (wins/losses/ties) and some examples of cases where they differ — concrete examples usually teach more than the percentage.

Offline vs. online A/B. Everything above is offline evaluation (pre-deployment). Online A/B (splitting real traffic between two prompt versions and measuring product metrics: resolution, human escalation, satisfaction) is the definitive validation, but requires feature flag infrastructure and volume. The mature flow: offline to iterate quickly and filter, online to confirm what matters.

The complete pipeline#

The module's mini-project brings the pieces together:

prompts/ (versionados) ──┐
                         ├─→ runner (ejecuta N×M) ─→ métricas programáticas
dataset de test ─────────┘                        ─→ juez LLM (si aplica)
                                                        │
                              informe: tabla por prompt, comparación,
                              regresiones vs versión anterior, coste de la eval

Existing tools that implement this pipeline (to familiarize yourself with them; in the labs we build it by hand precisely to understand it): promptfoo (YAML config, very straightforward for prompts), Inspect (AISI evals framework, Python), LangSmith / Braintrust / Langfuse (platforms with integrated datasets, runs, and judges), OpenAI Evals. In Module 5, evaluation is revisited as a continuous process in production.

How much it costs to evaluate (and why it doesn't matter)#

50 cases × 2 prompts × (1 generation + 2 judgments) ≈ 300 calls to cheap models ≈ pennies. Compared to the cost of deploying a worse prompt—or of arguing in a meeting about which one "looks better"—evaluation is the free part of the work. The real cost is building the dataset: that's why it is versioned and cared for as an asset.

Common errors#

  1. Evaluating on the same cases you used to iterate the prompt. This is manual overfitting to the "train set". Save a subset that you do not look at during development and use it only for the final comparison.
  2. Changing the prompt and dataset at the same time → incomparable numbers.
  3. Trusting a single pass of the judge in pairwise without reversing the order → position bias can decide your A/B test.
  4. Averaging everything into a single number → a prompt might raise the average while sinking a specific class. Look by-class/by-difficulty.
  5. Dataset only of easy cases → all prompts score 95% and you conclude that "the prompt doesn't matter".
  6. Not recording the model, temperature, and prompt version in the results → results are unreproducible a week later.
  7. Optimizing the metric instead of the objective (Goodhart's Law): if the judge rewards length, your prompts will "improve" by getting longer.

To go deeper#