Cost and latency budgets for agents
2 November 2026
Agents multiply cost and delay. One user question can turn into a dozen model calls and several tool calls. If nobody sets limits, the bill and the wait both grow without warning.
Measure per run
Track tokens, money and wall-clock time for each run, not just for each call. Then look at the median and the tail. The slowest and most expensive five percent of runs usually show where the design has a problem.
Set budgets
Pick a maximum for each of these and enforce it in code:
- number of steps
- total tokens
- total time
- number of tool calls of a given kind
When a budget runs out, stop with a clear message or hand off to a person. Do not let the agent continue silently.
Use the right model for each step
Not every step needs the largest model. Classifying a request, extracting a field or reformatting text can often use a smaller and faster model. Save the larger one for planning and hard reasoning. Check this with an evaluation set, so you know that quality holds.
Run independent work in parallel
If an agent needs three pieces of information that do not depend on each other, fetch them at the same time. This can cut latency a lot. Make sure the tools are safe to run in parallel, especially when they write.
Cache what repeats
Cache results of deterministic tool calls that are asked often. Where your provider supports prompt caching, keep the start of the prompt stable.
Make waiting visible
When a run takes a while, show progress. People tolerate a long wait much better when they can see that something is happening.