Build a repeatable test for an AI task I keep re-running

I keep running this task through AI: [THE TASK]. Every time a model changes I have no idea whether it got better or worse, so I want a test I can re-run.

What good looks like: [HOW YOU CURRENTLY JUDGE THE OUTPUT — even if it's "I just know"]
What bad looks like: [THE FAILURES YOU'VE ACTUALLY SEEN, not hypothetical ones]

Build me an eval:

1. THE TEST SET. Tell me how many real examples I need and — more importantly — what mix. It must include the ordinary case, the edge case I keep hitting, and at least one input that should produce a refusal or an "I don't know". A test set of only easy cases will pass forever and tell me nothing.

2. THE SCORING. Turn my vague "what good looks like" into criteria a second person could apply and reach the same score. Where something is genuinely subjective, say so and give a rubric rather than pretending it's binary.

3. THE FAILURE MODES TO WATCH SPECIFICALLY. From what I described as bad, name each distinct failure and how I'd detect it in an output. Some are visible instantly; some only show up when you compare against the source.

4. THE BASELINE. How I record today's results so a future run is comparable. Include what to note beyond the score — model version, date, settings, prompt version. A score with no version attached is not a baseline.

5. HOW OFTEN TO RUN IT, and the specific triggers that should force a run regardless of schedule.

6. THE HONEST LIMIT. Tell me what this eval will not catch, so I don't mistake a green result for a working system.

Keep it small enough that I'll actually run it. A rigorous eval nobody repeats is worth less than a rough one I run monthly.