Testing agents without flaky tests

Tests that call a real model are slow, cost money and give different answers on different runs. A test suite built only from them will be ignored soon after it is written.

Split the system

An agent has deterministic parts and a non-deterministic part. Test them separately.

• Tools. Normal functions. Test them like any other code.
• Graph or workflow logic. Routing, state updates and error handling are deterministic once the model output is fixed.
• The model. Only this part varies.

Replace the model in most tests

Write a fake model that returns scripted responses in order. Then you can test what the agent does when the model asks for a tool, returns bad JSON or never finishes.

class FakeModel:
    def __init__(self, responses):
        self.responses = list(responses)

    def invoke(self, messages):
        return self.responses.pop(0)

def test_agent_calls_search_tool():
    model = FakeModel([
        {"tool": "search", "args": {"q": "invoices"}},
        {"final": "Found 3 invoices"},
    ])
    result = run_agent(model, tools=[fake_search])
    assert result == "Found 3 invoices"

Test the failure paths

These matter most and are easiest to forget: a tool that times out, a malformed model response, a run that is resumed midway, an approval that is rejected.

Keep a few live tests

A small set of smoke tests against the real model is still worth having. Run them on a schedule or before a release, not on every commit, and accept that they may need an occasional rerun.

Quality of answers is a different question from correctness of code. For that you need evaluations, which is the next post.