What is a context window in AI? The limit that changes agent behavior

A context window in AI is the maximum amount of information a language model can consider in one request. It is measured in tokens, not pages or files, and it has to hold the instructions, conversation, retrieved documents, tool results and the answer being generated. When an agent fills that window, something must be removed, summarized or rejected. That decision changes behaviour.

• Context is what the model can see now. It is working material, not permanent knowledge.
• The window is a token budget. Every instruction, pasted document and tool result spends it.
• More context is not the same as better context. Relevant information can be buried inside material that technically fits.
• Memory sits outside the window. A system retrieves selected memories back into context when they are useful.
• Agents need a packing policy. Stable rules, current task state, recent evidence and tool results do not deserve equal retention.

If tokens are still fuzzy, start with what a token is in AI. The window is simply the largest token load the model accepts at once.

The plain definition

The model receives an ordered sequence of tokens and produces more tokens. The context window caps the combined sequence it can work over. Vendor APIs differ in whether output has its own separate allowance or shares the advertised limit, so the safe rule is to check the model documentation instead of subtracting from a marketing number.

Anthropic's context window documentation describes a rolling conversation: each turn includes the earlier messages again, and the model's output becomes input on the next turn: https://platform.claude.com/docs/en/build-with-claude/context-windows

A simplified request looks like this:

{
  "system": "Stable operating rules",
  "messages": [
    {"role": "user", "content": "The current task"},
    {"role": "assistant", "content": "Earlier work"},
    {"role": "user", "content": "A new correction"}
  ],
  "tools": ["search", "read_file", "run_test"]
}

The tool descriptions, earlier work and correction all consume context before the model reads one byte of a search result. An agent can therefore have a huge window and still waste most of it on permanent scaffolding.

What goes inside an agent's context window

A chat window usually contains a system instruction and conversation history. An agent window is busier. It can contain:

1. stable operating instructions;
2. the user's task and later corrections;
3. the current plan and completed steps;
4. tool definitions and schemas;
5. retrieved files or documents;
6. tool results, errors and command output;
7. summaries of earlier turns;
8. examples that define the expected output;
9. the tokens reserved for the next answer.

This explains why a "simple" agent session gets large quickly. My own main agent instruction file once reached 43 KB, roughly ten thousand tokens loaded before the task began. That fixed toll applied to a two-minute question and a two-hour build alike. The resulting CLAUDE.md audit moved specialist instructions behind on-demand skills instead of forcing every session to pay for them.

OpenAI exposes token-counting guidance because character counts and file sizes are only rough proxies for the real budget: https://developers.openai.com/api/docs/guides/counting-tokens

Context window vs memory

Context is the model's current desk. Memory is the filing system outside the room. The model cannot use a memory merely because it exists in a database. Something has to retrieve it, format it and put it back on the desk.

That distinction produces three common memory designs:

• Conversation history keeps recent turns directly in context. It is simple and gets more expensive on every turn.
• Summarized state replaces old detail with a compact record of decisions, facts and open questions. It saves space but can preserve a wrong summary.
• External memory or retrieval stores material outside the request and selects relevant pieces later. It scales, but retrieval quality becomes part of agent quality.

Calling all three "memory" hides the failure mode. In Claude Code that filing system is CLAUDE.md plus auto memory. A forgotten fact may have been trimmed, summarized incorrectly or never retrieved. Those require different fixes.

Why a larger context window can still perform worse

The most dangerous assumption is that anything inside the limit is equally available to the model. Research says otherwise. The "Lost in the Middle" study found that models often performed best when relevant information appeared near the beginning or end of a long input, and worse when it sat in the middle: https://arxiv.org/abs/2307.03172

Google's long-context guidance also warns that performance varies with the context, especially when a task requires finding multiple pieces of information. It recommends removing unnecessary tokens and placing the question after the supporting context: https://ai.google.dev/gemini-api/docs/long-context

This is not an argument against long windows. They unlock large-document work, many-shot examples and long-running agents. It is an argument against using the window as a bin. Capacity answers "does it fit?" Relevance answers "can the model find and use it?"

The packing policy I use for agents

I treat context as a loadout with four lanes.

Stable rules stay small. Safety boundaries, identity and output contracts belong near the front. Specialist procedures are loaded only when the task calls for them.

Current state stays explicit. Goal, completed steps, blockers and the next action should survive compaction. A transcript is not a state model.

Evidence stays close to the decision. The file content, API response or test result that supports the next action belongs in the current window. Old raw output can move to a trace store after its useful facts are extracted.

History is summarized by decision, not chronology. "We tried A, it failed because B, do not repeat it" is useful. Ten pages of the attempt are not.

A compact state object can be plain text:

goal: ship seven reviewed blog drafts
done:
  - keyword research approved
  - source set verified
constraints:
  - draft true
  - no deployment
next: validate MDX build
evidence:
  - test output path

The model does not need every sentence that produced this state. It needs the state and a way to inspect evidence when necessary.

Context limits, cost and latency

Input tokens are billed and processed on every request unless a provider offers a cache that applies. Repeating the same large instruction block therefore affects cost even if it never changes. Longer inputs also tend to increase time to first token. Google's documentation describes context caching as a way to reduce cost and latency for reused material, while noting that the information still occupies the logical context: https://ai.google.dev/gemini-api/docs/long-context

Caching is not compaction. It can make the same load cheaper; it does not make the load more relevant or increase the window. I use caching for stable, reused material and compaction for material the model no longer needs in full.

Frequently asked questions

What is a context window in AI?

A context window is the maximum number of tokens an AI model can consider in one request. It includes instructions, conversation history, retrieved content, tool results and the output allowance defined by the API.

Is a context window the same as AI memory?

No. Context is information available to the model in the current request. Memory usually lives outside the request and must be retrieved or summarized back into the context window before the model can use it.

What happens when the context window is full?

An API may reject the request, while a chat or agent system may trim old messages, summarize them or drop retrieved material. If trimming is silent, the agent can appear to forget rules or earlier decisions.

Does a bigger context window make an AI model better?

It gives the model more capacity, not guaranteed understanding. Irrelevant tokens increase cost and can bury important evidence, so selection and ordering still matter even when everything fits.

How do agents manage long context?

They keep stable rules small, store task state explicitly, retrieve evidence on demand, summarize old decisions and move full trajectories into an external trace store. The policy should be tested like any other agent component.

The useful mental model

The context window is not the model's brain and not its hard drive. It is the work surface for this turn. Good context engineering is deciding what earns space on that surface, what can be fetched later and what should be discarded. Once I started treating it as a loadout instead of a transcript, agent failures became easier to explain: the model either lacked a fact, lost a decision or was asked to find one useful needle in a pile I should never have loaded.