The Smallest Eval Harness

Previously I wrote about how I couldn’t understand the eval harness that I had built for Zoltar, my AI game master for solo tabletop roleplaying games (TTRPGs). I recently completed the first working version of its successor, ev2¹, the smallest and simplest eval harness that I could think of. Ev2 initially was built without any automated checkers and relied on hand-marking for scoring. It only tackles one question at a time, has no tags, and perhaps most importantly, doesn’t have the “provenance” field that so confused me in the original eval harness.

Fixtures

Before we get into the eval harness itself, let’s talk about fixtures. Fixtures are snapshots of game state that are read by the eval harness to test specific scenarios within the game. Unlike the eval harness, the process for capturing fixtures hasn’t changed. They’re constructed from game state in the database and stored as JSON files so they survive a wipe of the development database. A fixture for turn 23 includes all of the game state up to and including turn 22, plus the player’s input for turn 23.

The new harness adds fixtures constructed by hand from existing captured fixtures. Constructed fixtures allow us to investigate scenarios that didn’t arise in the original game session. For example, I’ve invested a lot of energy into the positional reasoning of the Warden (the model and prompt at the core of Zoltar). This comes up when the Warden has to narrate whether you have to change decks to get from engineering to the mess hall. By constructing fixtures, we can correct an error in an earlier turn to see its effect on later turns. The general discipline is the same as for any scientific experiment: only change one thing, keep everything else constant, and compare the results.

Ev2

Ev2 uses the JSON game state from the fixture to construct a scratch campaign in a database, and the player’s input is passed to the Warden. The Warden’s response is what gets graded. For each fixture, ev2 will repeat the process, and each repetition is recorded as a separate “rep.” We’re not interested in whether or not a given rep passes; we want to know the pass rate for a given fixture.

Unlike its predecessor, ev2 is designed to evaluate only one question at a time. Our first question is, “Does the Warden get descriptions of characters, places, and timelines correct against the seeded world facts and opening narration for labeled cases on at least 90% of reps per case?” Each question the harness addresses can have multiple fixtures.

Ev2 follows these steps:

1. Render something called a “case file,” which is just a Markdown file that includes the player input for the turn, the world facts at that point in the adventure, and the opening narration.
2. Seed a scratch adventure in the database.
3. Run the turn, make the Claude call, and record the Warden’s response as both JSON data and an easy-to-read Markdown file.
4. Remove the scratch adventure.
5. Repeat steps 2, 3, and 4 for each rep.
6. Repeat steps 1 through 5 for each fixture.
7. Create a marks.csv file for manual scoring. If a rep threw an error, error is recorded for that rep; otherwise, the field is blank for the grader to fill in.

marks.csv lets a human score the run by marking the output for each rep and recording pass, fail, or na for each one. A reporting utility is then used to generate a summary of the run, which will show the raw counts of pass, fail, na, and error for each fixture plus the rate of pass / (pass + fail) for each fixture. na and error results are omitted from the rate.

Each rep is scored based on a rubric, which is versioned for each question. The rubric for the first question started off just describing my process for hand-scoring the first run, but it evolved from there. I spent a lot of time agonizing over the phrase “back down” (and its lesser-used counterpart “back up”). Does it always indicate changing decks, or can you go back down to the mess hall from the crew berths without changing decks? It got so bad that in at least one grading session, I marked the phrase as both a pass and fail on different reps. The rubric eventually clarified the meaning well enough to use it for automated scoring, described below.

Hindsight being 20/20, I should have picked 88% as the pass rate target. A 90% target requires either all qualified reps to pass or 10 or more qualified reps. I normally only run 10 reps and would like to tolerate a single fail case. A 90% target means there can be no error or na cases. With 88%, 8 passes out of 9 reps exceeds the target, which allows for an error or na result. In other words, with a slightly lower target, one lost rep no longer sinks the case.

Judges

To keep ev2 simple, I hand-scored everything in the beginning. Hand-scoring was also necessary to evaluate an automated judge. The current judge is still a work-in-progress. Where the Warden uses Sonnet 5, the judge uses Opus 5.5, so we don’t have the problem of the same model generating and judging the output. The current judge’s scores can be used to tell if a rate improved, but they can’t tell us if a given run is “good enough.” For example, the judge can tell if a rate changed from 0.3 to 0.9, but it can’t tell you if the 0.9 score meets the 90% bar for Question 1.

This may seem counterintuitive, but I think about it like this: if the judge is biased and grades too strict or too loose, it does so consistently. In a comparison, that bias should cancel out. In practice, the evidence shows that it is consistently biased, at least in some cases. If both runs contain the same kinds of narration, with the same kinds of phrasing, then the judge’s bias is more likely to be consistent between the runs and cancel out. It takes some more digging to figure out if we can even trust the comparison, which is a reason why evals can be a pain in the neck and why the judge is a work-in-progress.

For the judge to be able to determine whether a rate is good enough, I need to figure out how often I disagree with myself. Some of this work is already done. In one measure, my own revision rate on second look was about 5%. In a different measure, the judge and I agreed on 62 out of 65 marks. The three disagreements turned on a phrase that I marked as both pass and fail in one sitting². What I’m waiting on now is rescoring a previous run after allowing enough time to pass so that the second score is effectively as blind as the first. Then I’ll know how inconsistent I am in my own scoring. If I only agree with myself 90% of the time, the judge can’t be expected to agree more often than that. The self-agreement rate defines a ceiling. Once I have established that ceiling, I will know if the current judge is good enough or if it needs updates to reach that level of performance.

Ev2 in Action

The new eval harness has already proven to be a very useful tool for improving Zoltar. By using constructed fixtures, I was able to show that a persistent error in the Warden’s spatial reasoning was caused by an incorrect upstream narration. In the constructed fixture, I fixed the upstream narration, and the pass rate improved from 1 of 10 to 8 of 9³. Trying to solve that problem with prompt changes and adding corrections on top of the error was ineffective.

In a real session, however, you can’t go back and fix past mistakes. The Warden needs to get it right the first time, but the Warden’s own spatial reasoning capabilities are suspect. I needed a way to increase the odds of the Warden getting a spatial problem correct. I prototyped a new deterministic route tool that the Warden can use in place of its own reasoning, and the tool increased the judged pass rate from 13 of 29 to 27 of 29. As discussed above, I can’t say if that’s good enough, but I can definitely say it’s an improvement.

Look at me, learning stuff!

I previously wrote that human documentation needs to be a part of the agentic engineering process. This article just goes to show the benefit of this practice. In order to write it, I really had to drill down into the difference between using the automated judge to measure whether a rate improved and using the automated judge to tell whether a rate is good enough. It was a bit of a process, and I had Claude Code there to help me through it. And when I got to writing about the improved pass rate of the route tool, I was able to sail right through it. By writing about it, I’m building my own understanding of how the code works.

Footnotes

1. “Ev2” is short for eval-v2. Super creative, I know. ↩

2. The phrase in question being “back down,” as discussed above. ↩

3. 8 of 9 is an 89% pass rate. This is the case that made we wish I’d set a pass rate target of 88%, as discussed above. ↩