Keeping agent services stateless

A stateless service keeps no important data in its own memory between requests. All the state it needs is stored somewhere else. For agent services this is one of the most useful rules to follow.

Why it matters

If a run lives in the memory of one process, then:

• a deploy or a crash loses the run
• the same run must always reach the same instance
• scaling out is awkward, because instances are not interchangeable

If the state lives in a database, any instance can pick up any run. Instances become disposable.

What to store externally

• the conversation or message history
• the position in the workflow, meaning which steps are done
• tool results that later steps depend on
• pending approvals and their status

A simple shape

1. A request starts a run and writes its initial state to storage.
2. A worker loads the state, does one step, and writes the result back.
3. The next step can run on the same worker or a different one.

Each step begins by reading and ends by writing. The worker holds nothing in between.

Trade-offs

Reading and writing state at every step adds latency and load on the store. Keep the stored state small, and store large payloads by reference. In return you get recovery after failures, simple deploys and straightforward scaling.

This is the same idea as the checkpointing from earlier posts, applied at the level of the whole service.