If you have ever pasted a long document into an AI chat tool and found that, twenty messages later, it seemed to have “forgotten” something you told it at the start, you have already bumped into the context window. It is one of the least talked-about ideas in AI, and one of the most useful to understand, because it quietly shapes what these tools can and can’t do for you.
In this post I want to explain what a context window is in plain English, why it is measured in tokens, what happens when it fills up, and why a bigger window isn’t automatically better. No maths, I promise — just the mental model I wish someone had given me early on.
The model’s working memory
The simplest way to think about a context window is as the model’s working memory. It is everything the model can “see” at the moment it writes a response: your instructions, the conversation so far, any documents you have attached, and the response it is in the middle of writing. Anthropic’s documentation describes it exactly this way — the context window is all the text a language model can look back on and reference when generating new text, plus the new text it generates, and it is distinct from the huge body of data the model was originally trained on.[1]
That last point matters. A large language model has two very different kinds of “knowledge”:
- What it learned in training — general knowledge baked into the model months before you ever used it. You can’t change this.
- What is in its context window right now — the specific information you give it in this conversation. This is the part you control.
When an AI tool answers a question about your contract, your sales figures or your policy document, it can only do so because that material has been placed in the context window. Once something falls outside the window, the model simply can’t see it any more.
Measured in tokens, shared by input and output
Context windows are measured in tokens, not words or pages. If you haven’t read it yet, my earlier post on what tokens are explains them properly, but the short version is that a token is a chunk of text, often a piece of a word. In English, a token works out to roughly three-quarters of a word on average.
The key thing most people miss is that the window is shared. Your prompt, any attached files, the whole back-and-forth of the conversation and the model’s reply all have to fit in the same space.[1] In a chat, every new message gets added to the pile, so a long conversation steadily uses up the window even if each individual message is short.
| What goes into the window | Example |
|---|---|
| System instructions | “You are a helpful assistant for an accounting firm…” |
| Conversation history | Every earlier question and answer in the chat |
| Attached material | A PDF, a spreadsheet export, a pasted email thread |
| The model’s response | The answer it is writing right now |
Modern windows are genuinely large. Google’s Gemini documentation, for example, describes models with context windows of a million tokens or more — enough to hold very long documents, large codebases or hours of transcribed audio in one go.[2] A few years ago, a few thousand tokens was normal. Exact sizes change with almost every model release, so I won’t quote numbers for specific products here; check the provider’s documentation for the model you are using.
What happens when the window fills up
Every window has a limit, and different tools handle hitting it differently. Depending on the product, you might see:
- An error telling you the input is too long.
- Older messages quietly dropped or summarised to make room, which is often the cause of that “it forgot what I said earlier” feeling.
- A cut-off response, because there wasn’t enough room left for the model to finish its answer.
Anthropic’s documentation describes both ends of this: each message and response accumulates in the window as a conversation goes on, an input that is already too big is rejected with a “prompt is too long” error, and a response that runs into the limit simply stops.[1] Chat apps built on top of these models often hide that from you by trimming or summarising older turns. Either way, the practical lesson is the same: in a long session, important details from the beginning are at risk. If something really matters, restate it, or start a fresh conversation with a clean summary.
For long pieces of work, I keep a short “brief” — the goal, the key facts and the constraints — and paste it at the top of a new chat whenever a conversation starts to drift. It costs a few hundred tokens and saves a lot of frustration.
Bigger isn’t always better
It is tempting to think that if a model can take a million tokens, you should just give it everything. In practice, there are three reasons to be more thoughtful.
1. Models don’t pay equal attention to everything
Researchers at Stanford and elsewhere tested how well language models use long inputs and found a striking pattern: performance was often highest when the relevant information sat at the beginning or end of the input, and dropped noticeably when it was buried in the middle.[3] They called the paper “Lost in the Middle”, which is a nice summary of the risk. If you have one critical instruction, don’t hide it in paragraph forty of a long prompt.
2. The advertised size isn’t always the useful size
A benchmark called RULER was built to test whether long-context models really perform well across their full advertised window. The researchers found that many models handled simple “find the needle in the haystack” tests well, but almost all of them showed large performance drops on harder tasks as the input got longer. Every model tested claimed a window of at least 32,000 tokens, yet only half could maintain satisfactory performance at that length.[4] In other words, “fits in the window” and “understood well” are not the same thing.
3. Every token costs money and time
Most AI services charge per token, and longer inputs take longer to process. Sending a 300-page manual with every question works, but it is slow and expensive at scale. I unpack the cost side of this in the hidden risk of Pure AI products. Later in this series I’ll also cover a technique that solves much of this problem by fetching only the relevant pages of a document instead of sending all of it.
Rule of thumb: give the model what it needs to do the job well, put the most important instructions at the start (and repeat them at the end for long inputs), and leave out the rest.
What I want you to take away
- The context window is the model’s working memory. It is everything the model can see right now, separate from what it learned in training.
- It is measured in tokens and shared. Your instructions, the conversation history, attachments and the reply all compete for the same space.
- When it fills up, something gives. Older details get dropped, responses get cut off, or you get an error. Restate what matters in long sessions.
- Position matters. Models tend to use information at the start and end of an input better than information in the middle.
- Bigger isn’t automatically better. Long inputs can reduce quality and always increase cost. Curate what you send.
Working out where AI fits in your business?
I help Australian teams cut through the hype and pick AI projects that actually pay off. If you'd like a second opinion on an idea, I'm happy to talk it through.
Get in touch →References
- Anthropic. Context windows. Claude Developer Platform documentation. https://platform.claude.com/docs/en/build-with-claude/context-windows
- Google. Long context. Gemini API documentation. https://ai.google.dev/gemini-api/docs/long-context
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. https://arxiv.org/abs/2307.03172
- Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654. https://arxiv.org/abs/2404.06654