---
title: The judge, and the statistics that
judge the judge
subtitle: An LLM-as-a-judge QA tool, grader
calibration, and a statistical
re-baseline for an RL environment program
slug: llm-judge-and-rebaselining
hook: One task went from 3/20 to 20/20 after a
platform redeploy. Better agent,
or broken task? I ran the numbers before
anyone argued about it.
role: AI Engineer, RL environments and
evaluation
employer: Turing (contract, remote)
client: An enterprise data & AI platform
company
period: May – Aug 2026
tracks:
- evaluation
- data-science
- agentic-genai
tags:
- LLM-as-a-judge
- eval statistics
- Fisher's exact test
- grader calibration
- QA automation
metrics:
- value: ≈89%
label: of the first delivery's tasks carry
a verdict from the QA judge I built
(92 of 103)
when: May – Jun 2026
- value: p = 2.6 × 10⁻⁸
label: One task's jump from 3/20 to 20/20
passes, by Fisher's exact test
when: Jul – Aug 2026
- value: 3 of 4
label: Re-baselined tasks with a
significant shift; the fourth (3/10 →
8/10, p =
0.07) was not
when: Jul – Aug 2026
---
## Context
Same RL-environment program as case studies 01
and 02, earlier in its life. Every eval
team eventually faces two questions:
1. **Is the grader right?**
2. **When a number moves, did the thing we
measure change, or did the ruler?**
Between May and August 2026 I built tooling
for the first question and ran the
statistics for the second.
## The problem
Graders fail in two directions. One that fails
a correct answer teaches the model the
wrong lesson. One that passes a wrong
answer hides the failure. Both look fine
on a dashboard.
Pass rates fail too. With 10 or 20 runs per
task, a "big jump" can be noise, and real
improvements can hide inside noise. When a
platform redeploy sent some delivered
tasks' pass rates soaring, the team needed
to know which tasks still worked as
intended before anyone retired or rewrote
them.
## Constraints
- **Runs are expensive,** so samples are
small: 10 or 20 per task per model.
- **The client re-graded everything with its
own judge.** Our judge and theirs had to
agree.
- **Delivered tasks are load-bearing.** You
don't rewrite one because of a hunch.
## What I built
**1. An LLM-as-a-judge QA tool (May – Jun
2026).** An internal QA judge built on a
frontier LLM's API: about 1,800 lines of
Python, packaged so other authors could
run it. For each new task package it runs:
- duplicate-task detection against everything
already delivered;
- a live catalog collision check, so a new
task's tables can't collide with existing
ones;
- format, taxonomy and review checks;
- an auto-generated delivery row and verdict
file.
**2. Graders that are tested before they
judge.** Every grader I shipped was
self-tested against the gold answer, two
or three genuinely different correct
answers, and every known wrong answer.
Judge rubrics were written asymmetrically:
sharp about the defect, loose about the
route, five checks or fewer. From the
third phase of the program, I reproduced
every run's grade with the client's own
judge.
One example. A probe run reached the correct
answer but was graded a non-pass. The
grader had read the agent's description of
its own verification step as an admission
of a shortfall. We fixed the rule (a
verification sentence is a check, not a
confession), re-ran the grader's
self-tests and re-graded all 13 graded
runs of that task. Only the misgraded run
moved.
**3. A statistical re-baseline (Jul – Aug
2026).** After the client redeployed its
agent, pass rates on some delivered tasks
jumped. One went from 3/20 to 20/20. The
project lead asked for a re-baseline. I
re-ran the baselines of 4 tasks (50 runs),
checked the workspace for drift, and
tested each task's shift with Fisher's
exact test.
- Three of the four shifts were significant,
with p-values from 2.6 × 10⁻⁸ (3/20 →
20/20) to 0.011. One of the three, 2/10 →
10/10, came in at p = 7.1 × 10⁻⁴.
- The fourth, 3/10 → 8/10, looked like a win
and wasn't one: p = 0.07.
- The workspace had not drifted. So the gains
were real: the agent had improved, not the
tasks broken.
- I flagged which traps needed recalibration,
and the lead decided from evidence rather
than from a hunch.
## How it fits together
Two loops.
**The QA loop** runs on the way in. A new task
package goes to the judge, which checks it
against every delivered task, the live
catalog, and the format and taxonomy
rules, then writes a verdict file and the
delivery row.
**The trust loop** runs when a number moves.
Re-run the baselines, test each shift, and
check the environment for drift, in
parallel. Then decide: a real gain means
the trap needs recalibrating; a broken
task means fixing the task.
## Stack
- **Judge:** a frontier LLM through its Python
SDK, structured verdict files,
LLM-as-a-judge rubrics.
- **Statistics:** Fisher's exact test (SciPy),
pass-rate calibration bands, run
distributions.
- **Data:** live catalog queries against the
data platform, SQL, Python, pandas.
- **Delivery:** Git and GitHub, packaged CLI
tooling for other authors.
## Results
- **The judge's verdicts appear on 92 of the
103 tasks** in the first delivery repo.
- **A misgrade caught and fixed** without
disturbing any correct grade: 13 runs
re-graded, one moved.
- **The client's judge reproduced every
shipped run on both model arms,** so
delivery had no regrade surprises.
- **A re-baseline that separated signal from
noise:** 3 of 4 shifts significant, one
not, and no workspace drift.
## What I'd do differently
- **Write the protocol before rerunning
anything.** The test, its sidedness, the
threshold and the run counts belong in a
short standing document. Choosing them
after seeing the data is the easiest way
to fool yourself, and the 3/10 → 8/10 case
is exactly where it happens.
- **Re-baseline on a schedule.** We
re-baselined because of a surprise. A
small canary set re-run every week would
have caught the shift the day it happened.
- **Calibrate the judge against human labels
first.** A small human-labeled agreement
set, measured before launch, finds
misgrades earlier than probe runs do.
- **Show intervals next to pass rates.** With
10 runs, the uncertainty is wide. Putting
the interval on the dashboard ends a lot
of arguments before they start.
I wrote up the method in the essay "Is that
eval improvement real?".
## What I left out
This is client work under NDA. Not included,
on purpose:
- the client's name and product, and which
tasks moved;
- the judge's prompts and the client's judge
prompt;
- trap designs, the taxonomy and task text;
- model names and the details of the platform
redeploy;
- the delivery repository and its contents.
## How to verify
- **Recompute it yourself.** Fisher's exact
test on the before/after counts reproduces
every p-value here. In Python,
`scipy.stats.fisher_exact([[3, 17], [20,
0]])` gives p ≈ 2.6 × 10⁻⁸, and `[[3, 7],
[8, 2]]` gives p ≈ 0.07.
- **LinkedIn, Turing experience entry** (Mar
2024 – present): LLM-as-a-judge grading
over 20+ runs on two frontier models.
- **Employment check:** Turing, official title
Consultant, contract, Mar 2024 – present.
- **In an interview:** a walk-through of the
method, within NDA limits.
- **Code:** walkthroughs available on request.
Case study 04 · Turing (contract, remote) · May – Aug 2026
The judge, and the statistics that judge the judge
An LLM-as-a-judge QA tool, grader calibration, and a statistical re-baseline for an RL environment program
One task went from 3/20 to 20/20 after a platform redeploy. Better agent, or broken task? I ran the numbers before anyone argued about it.
✓ Compiled from this page's own markdown.
Context
Same RL-environment program as case studies 01 and 02, earlier in its life. Every eval team eventually faces two questions:
- Is the grader right?
- When a number moves, did the thing we measure change, or did the ruler?
Between May and August 2026 I built tooling for the first question and ran the statistics for the second.
The problem
Graders fail in two directions. One that fails a correct answer teaches the model the wrong lesson. One that passes a wrong answer hides the failure. Both look fine on a dashboard.
Pass rates fail too. With 10 or 20 runs per task, a "big jump" can be noise, and real improvements can hide inside noise. When a platform redeploy sent some delivered tasks' pass rates soaring, the team needed to know which tasks still worked as intended before anyone retired or rewrote them.
Constraints
- Runs are expensive, so samples are small: 10 or 20 per task per model.
- The client re-graded everything with its own judge. Our judge and theirs had to agree.
- Delivered tasks are load-bearing. You don't rewrite one because of a hunch.
What I built
1. An LLM-as-a-judge QA tool (May – Jun 2026). An internal QA judge built on a frontier LLM's API: about 1,800 lines of Python, packaged so other authors could run it. For each new task package it runs:
- duplicate-task detection against everything already delivered;
- a live catalog collision check, so a new task's tables can't collide with existing ones;
- format, taxonomy and review checks;
- an auto-generated delivery row and verdict file.
2. Graders that are tested before they judge. Every grader I shipped was self-tested against the gold answer, two or three genuinely different correct answers, and every known wrong answer. Judge rubrics were written asymmetrically: sharp about the defect, loose about the route, five checks or fewer. From the third phase of the program, I reproduced every run's grade with the client's own judge.
One example. A probe run reached the correct answer but was graded a non-pass. The grader had read the agent's description of its own verification step as an admission of a shortfall. We fixed the rule (a verification sentence is a check, not a confession), re-ran the grader's self-tests and re-graded all 13 graded runs of that task. Only the misgraded run moved.
3. A statistical re-baseline (Jul – Aug 2026). After the client redeployed its agent, pass rates on some delivered tasks jumped. One went from 3/20 to 20/20. The project lead asked for a re-baseline. I re-ran the baselines of 4 tasks (50 runs), checked the workspace for drift, and tested each task's shift with Fisher's exact test.
- Three of the four shifts were significant, with p-values from 2.6 × 10⁻⁸ (3/20 → 20/20) to 0.011. One of the three, 2/10 → 10/10, came in at p = 7.1 × 10⁻⁴.
- The fourth, 3/10 → 8/10, looked like a win and wasn't one: p = 0.07.
- The workspace had not drifted. So the gains were real: the agent had improved, not the tasks broken.
- I flagged which traps needed recalibration, and the lead decided from evidence rather than from a hunch.
How it fits together
Two loops.
The QA loop runs on the way in. A new task package goes to the judge, which checks it against every delivered task, the live catalog, and the format and taxonomy rules, then writes a verdict file and the delivery row.
The trust loop runs when a number moves. Re-run the baselines, test each shift, and check the environment for drift, in parallel. Then decide: a real gain means the trap needs recalibrating; a broken task means fixing the task.
Stack
- Judge: a frontier LLM through its Python SDK, structured verdict files, LLM-as-a-judge rubrics.
- Statistics: Fisher's exact test (SciPy), pass-rate calibration bands, run distributions.
- Data: live catalog queries against the data platform, SQL, Python, pandas.
- Delivery: Git and GitHub, packaged CLI tooling for other authors.
Results
- The judge's verdicts appear on 92 of the 103 tasks in the first delivery repo.
- A misgrade caught and fixed without disturbing any correct grade: 13 runs re-graded, one moved.
- The client's judge reproduced every shipped run on both model arms, so delivery had no regrade surprises.
- A re-baseline that separated signal from noise: 3 of 4 shifts significant, one not, and no workspace drift.
What I'd do differently
- Write the protocol before rerunning anything. The test, its sidedness, the threshold and the run counts belong in a short standing document. Choosing them after seeing the data is the easiest way to fool yourself, and the 3/10 → 8/10 case is exactly where it happens.
- Re-baseline on a schedule. We re-baselined because of a surprise. A small canary set re-run every week would have caught the shift the day it happened.
- Calibrate the judge against human labels first. A small human-labeled agreement set, measured before launch, finds misgrades earlier than probe runs do.
- Show intervals next to pass rates. With 10 runs, the uncertainty is wide. Putting the interval on the dashboard ends a lot of arguments before they start.
I wrote up the method in the essay "Is that eval improvement real?".
What I left out
This is client work under NDA. Not included, on purpose:
- the client's name and product, and which tasks moved;
- the judge's prompts and the client's judge prompt;
- trap designs, the taxonomy and task text;
- model names and the details of the platform redeploy;
- the delivery repository and its contents.
How to verify
- Recompute it yourself. Fisher's exact test on the before/after counts reproduces every p-value here. In Python,
scipy.stats.fisher_exact([[3, 17], [20, 0]]) gives p ≈ 2.6 × 10⁻⁸, and [[3, 7], [8, 2]] gives p ≈ 0.07.
- LinkedIn, Turing experience entry (Mar 2024 – present): LLM-as-a-judge grading over 20+ runs on two frontier models.
- Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
- In an interview: a walk-through of the method, within NDA limits.
- Code: walkthroughs available on request.