Evaluating LLM pipelines with small datasets

You do not need a large benchmark to evaluate an LLM pipeline. A small set of real cases, run the same way every time, already tells you whether a change helped or hurt.

Start with real examples

Collect thirty to fifty cases from actual usage. Include easy ones, hard ones and a few that previously failed. For each, write down what a correct result looks like.

Pick checks you can automate

The easiest checks are exact ones:

• Does the generated SQL return the expected rows?
• Is the output valid JSON that matches the schema?
• Did the agent call the expected tool?
• Does the answer contain the required fact?

For open-ended answers you can use a model as a judge, but treat it as a rough signal and read some of the results yourself.

Run it on every change

Change a prompt, a model or a tool description, then run the set and compare pass rates. This is what stops a fix for one case from quietly breaking five others.

Look at the failures, not just the score

A drop from 80 to 76 percent says little. The four cases that now fail say a lot. Group failures by cause, such as wrong tool, bad retrieval or bad formatting, and fix the largest group first.

Watch for overfitting

If you keep tuning the prompt against the same fifty cases, you will eventually fit those cases rather than the real problem. Add new cases regularly, and keep some aside that you never tune against.

Small and honest beats large and untouched.