Managing the context window
27 October 2026
Every model call has a limit on how much text it can take, and every token in it costs time and money. In a long agent run, the history grows with each step. If you do nothing, the run eventually fails or gets expensive.
Know what is taking space
Measure it. Count tokens for the system prompt, the history and each tool result. The largest part is usually tool output, not the conversation.
Trim tool results
Return only the fields the model needs. If a tool can return a thousand rows, return a summary and the first few rows, with a way to ask for more. Do this in the tool itself so that the full payload never enters the context.
Summarize old turns
When the history gets long, replace the oldest part with a short summary and keep recent messages as they are. Keep facts the task depends on, such as ids, decisions and constraints, because summaries tend to lose them.
Keep stable text at the start
Put instructions that never change at the beginning of the prompt. Some providers cache a repeated prefix, which lowers cost and latency when the start of the prompt is identical across calls.
Retrieve instead of stuffing
Do not paste whole documents into the prompt. Search for the relevant passages and include only those. Fewer, better passages usually give better answers than many mixed ones.
Set a budget
Decide a token budget per step and per run. When a call would exceed it, compress or stop deliberately, instead of letting the API reject the request.