A cheap eval loop that catches 80% of regressions

You do not need a serious eval framework to catch most prompt regressions. You need thirty test cases, a plain CSV, and about an hour of setup.

I keep meeting teams who don't have an eval loop for their prompts because they think an eval loop is a big engineering project. It isn't. Or, more precisely: the version of it that catches most of the pain is embarrassingly small, and you can build it in a lunch break.

Here is the version I use for personal projects. It is not what I would ship for a regulated production system. It is what I use to stop shipping obviously-worse prompts to myself.

The whole loop, in five moves

1. Write 30 short test cases in a CSV: id, input, must_include, must_not_include, notes.
2. Write a 40-line script that runs each input through the current prompt and records the output.
3. For each row, mark a pass/fail: pass = every must_include substring appears, no must_not_include substring appears.
4. Save the pass/fail table and the raw outputs into a run folder named by date and prompt version.
5. When you change the prompt, run again and diff the two run folders.

Why 30 cases

Because you will actually write 30 cases. You will not actually write 300. Thirty is enough to catch structural regressions (empty outputs, wrong format, wrong language, the "as an AI assistant" preamble sneaking back in). It is not enough to catch subtle quality drift, but you catch that in code review anyway.

Thirty test cases you actually run beats three hundred test cases you meant to write.

Why substring matching, not judge-model grading

Judge-model grading is fine, but it introduces two problems on day one: it costs money, and it drifts. A boolean substring test is free, deterministic and lasts forever. For structural regressions — which are what you actually care about at this stage — it is more than enough.

Concretely, this is what a row in my CSV looks like:

id, input, must_include, must_not_include, notes
r01, "3 bullet summary of ...", "•|•|•", "As an AI", 3 bullets exactly
r02, "extract emails from ...", "@", "sure!|here are", must find at least one email
r03, "translate to fr: hello", "bonjour", "hello|hi", basic FR
r04, "empty input case: ''", "", "traceback|error", must not crash

The | is a pipe-separated OR list. must_include: "•|•|•" means "at least three bullet characters." Not elegant. Very effective.

The diff step is the whole point

The magic of this loop is not the individual runs. It is the diff. When you change a prompt and re-run, you don't ask is my new prompt good? You ask did anything that used to pass now fail? That question is boolean, and you can answer it in ten seconds.

Every time I skip this step because I am in a hurry, I regret it. Every time. The number of "small" prompt changes that silently break case #17 is embarrassing.

What this does not catch

• Subtle quality regressions ("outputs got wordier"). You need a judge model or a human for these.
• Hallucinated facts that pass all your substring rules. Same.
• Latency and cost regressions. Add another column and log them.
• Format drift over time in the underlying model. You will only notice these when the whole set gets worse; add a "date of last full run" note and refresh monthly.

A note about honesty

The most common mistake I see people make with a light eval loop is to treat the pass rate as the score. It isn't. The score is the number of rows that used to pass and don't any more. If your suite is at 27/30 pass rate and it stays at 27/30 after your change, that's not a good result — check which three failed.