Context window

How it works

Everything a model works with on a given request has to fit in the window: the system prompt, the conversation so far, retrieved documents, tool definitions, tool results and the answer it writes. A token is a piece of a word; in English, 1,000 tokens is roughly 750 words. Frontier models in late 2026 accept hundreds of thousands of tokens, and several accept 1 million or more.

The model keeps no memory between requests. A chatbot that "remembers" earlier messages is resending them each time, which is why long conversations get slower and more expensive.

How businesses use it

Large windows make some jobs simple: compare 2 contracts in full, summarize a long call transcript, or check a 200-page RFP against a checklist in 1 pass. For one-off analysis that is often the right approach.

For an assistant that answers questions across thousands of documents, retrieval is still the better design. RAG sends only the relevant passages, so each request costs less and returns faster.

Context also limits AI agents. Every tool definition and tool result takes up space, so an agent with 40 tools and a long task can fill its window before it finishes. Well-built custom agents trim old tool output and summarize as they go.

Where it goes wrong

A bigger window does not mean the model reads all of it equally well. The Lost in the Middle (opens in a new tab) study (Liu and colleagues, 2023) found that models used information at the start and end of a long input better than information buried in the middle. Test your own long-document tasks before trusting the advertised size.

Cost is the other trap. You pay for every input token on every request, so a prompt that pastes a 300-page manual into each question gets expensive quickly. Send only what the task needs, and keep standing instructions short enough that they do not crowd out the documents.