Context windows: what the model can see

Where you are

The practice pages so far have been short: a few hundred words each. Real web pages aren't always like that. A town's minutes page can hold a year of meetings, and the one fact you need may be in a single line near the top.

This lesson is about what happens when the page is long. It's the most common way a model gives a wrong answer for a reason that has nothing to do with how smart it is: it simply never saw the part that mattered.

The window

A model doesn't read your prompt the way you read a page, top to bottom, keeping what it needs. It takes in a fixed number of tokens at once, its context window, and works with all of them together. The window holds everything: the hidden system message, your instructions, the page, and the answer as it's written.

Two numbers matter, and they're easy to mix up:

• The model's maximum. In Lesson 2, ollama show listed context length 32768. That's the most qwen2.5:3b was built to handle.
• The window Ollama actually uses, set by an option called num_ctx. It can be smaller than the maximum, to save memory. ollama ps shows it in the CONTEXT column.

Ollama picks the default window from how much graphics memory your machine has. According to its documentation, under 24GB gets 4,096 tokens, 24 to 48GB gets 32,768, and more gets bigger still, up to the model's maximum. The laptop in these lessons has 48GB, which is why ollama ps showed 32,768. On most laptops, the default is 4,096. Keep that number in mind for what comes next. Older versions of Ollama used other defaults, so check ollama ps on your own machine rather than assuming.

You can set num_ctx yourself for each request, in the options from Lesson 4, next to the temperature:

"options": {"temperature": 0, "num_ctx": 8192},

A long page

The practice set now has one long page, long/larch-grove.html: a year of council meeting minutes for another made-up town. It's about 30,000 characters of text. The mayor's name, Nadia Brennan, appears exactly once, in the second line. After that, the minutes only ever say "the Mayor".

context.py asks the narrow prompt from Lesson 5 about that page, with four window sizes. For each, it prints how many tokens the model actually read (prompt_eval_count, from Lesson 4), how much memory Ollama was using, and the answer:

def run(label, prompt, num_ctx):
    r = requests.post(f"{OLLAMA}/api/generate", json={
        "model": MODEL,
        "prompt": prompt,
        "stream": False,
        "options": {"temperature": 0, "num_ctx": num_ctx},
    }, timeout=300)
    r.raise_for_status()
    data = r.json()
    ps = requests.get(f"{OLLAMA}/api/ps", timeout=10).json()["models"]
    ...

/api/ps is the API version of ollama ps. Pull the latest practice set (git pull), then run it:

python3 context.py

The page is 30,462 characters of text.

instructions first     num_ctx   2048  read  1026 tokens  memory  2.1 GB  answer: Based on the provided text, here is a summary of the key points from t...
instructions first     num_ctx   4096  read  2050 tokens  memory  2.1 GB  answer: Based on the provided text, it appears that this passage is a series o...
instructions first     num_ctx   8192  read  6622 tokens  memory  2.3 GB  answer: Nadia Brennan
instructions first     num_ctx  32768  read  6622 tokens  memory  2.8 GB  answer: Nadia Brennan

instructions last      num_ctx   2048  read  1026 tokens  memory  2.1 GB  answer: NONE

The focused page is 1,587 characters.
focused page           num_ctx   2048  read   371 tokens  memory  2.1 GB  answer: Nadia Brennan

Start with the first four lines. The whole prompt is 6,622 tokens. With a window of 8,192 or more, all of it fits, and the answer is right. With 2,048 or 4,096, it doesn't fit (and 4,096 is the default on most laptops), and look what the model read: 1,026 tokens and 2,050, about half of each window. Most of the prompt never reached the model.

The silent cut

The answers tell you which part went missing. With the instructions at the start of the prompt, the model didn't answer the question at all: it summarized the minutes. It never saw the question, so it did what a model does with a long document and no instructions.

So Ollama kept the end of the prompt and dropped the beginning, and the beginning is where the question was. The second run confirms it. With the instructions moved to the end, they survived the cut, but the mayor's name, near the top of the page, didn't. The model answered NONE, which was honest: in the part it read, there was no name.

Three things make this dangerous:

• There's no error. The reply looks like any other: done_reason is still "stop", and nothing says "I only read part of this".
• It drops the part you'd least expect. Most prompts put the instructions first. That's exactly the part that goes.
• The answer can still look plausible. A summary is obviously wrong for code expecting a name. But a model that kept the question and lost the facts might have invented a name instead of saying NONE, and that would look exactly like a right answer.

The one clue is prompt_eval_count. If it's much smaller than your prompt should be, the prompt was cut. A rough rule for English text: about 4 to 5 characters per token. This page was 30,462 characters and 6,622 tokens, so a reading of 1,026 tokens was a clear sign that most of it was missing.

Switch the settings below to see what reached the model each time:

Bigger windows cost memory

The obvious fix is a big window, and 8,192 was enough here. But look at the memory column: 2.1GB at the small sizes, 2.3GB at 8,192 and 2.8GB at 32,768, for exactly the same answer as 8,192.

That's because Ollama sets aside memory for the whole window when it loads the model, whether you use it or not. Lesson 2's 2.8GB was the 32,768-token window. On a laptop, that's the difference between a model that fits alongside your other apps and one that doesn't; with bigger models, the window's share grows. Bigger windows are also slower to fill: a model reading 6,000 tokens takes longer than one reading 400.

So the window should be big enough for your largest prompt, plus room for the answer, and not much bigger.

Three fixes

The results show three ways to handle a long page, each with a trade-off:

1. A window big enough for the whole prompt. 8,192 worked here. It's the simplest fix, but it costs memory, and the next page could be longer still.
2. Instructions at the end. If anything gets cut, it's the start of the page, not the question, so a too-long page gives NONE instead of nonsense. It's a cheap safety net, not a fix: the answer was still lost.
3. Send less. context.py kept only the lines that mention the mayor: 1,587 characters, 371 tokens. The smallest window, the smallest memory, and the right answer.

The third is the one that scales. Most of a long page has nothing to do with your question: 6,000 tokens of sidewalks, snow removal and library roofs. Cutting the page down to the part that matters makes the model faster, cheaper to run and less likely to get distracted.

Its trade-off is that the filter can drop what you need. context.py's filter keeps lines containing "Mayor". Run that same filter on the ten practice pages, and it loses the answer on two of them. On Alder Creek, the name and the word "Mayor" are in different cells of a table, so they end up on different lines. On Heron Point, the name sits on its own line, above "Mayor of Heron Point". A filter that's too strict hands the model a page without the answer, and the model will rightly say NONE. So the filter needs testing too, on every page, the same as a prompt.

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

It worked on short pages. A script that extracts invoice totals works perfectly on one-page invoices. On a twenty-page contract, it returns a paragraph describing the contract. Nothing in the code changed. What happened? The long document didn't fit in the context window, so the start of the prompt, with the instructions, was cut. The model never saw the question. Checking prompt_eval_count would have shown it.

A bigger window for everything. Someone sets num_ctx to the model's full 32,768 for every request, "to be safe". Their prompts are never more than 3,000 tokens. What does it cost them? Memory, for no benefit: Ollama sets aside room for the whole window when it loads the model. In this lesson, that was 2.8GB instead of 2.3GB, for the same answers.

The filter that hid the answer. To keep pages short, a team sends the model only the lines containing the word "Mayor". It works on most towns, but returns NONE for several that clearly have a mayor. Why? Their filter is too strict: it misses names that sit on a different line from the title, like a table cell or a signature. Shrinking the input is a good idea, but the filter needs testing on every page, just like a prompt.

Where you are now

A model works with a fixed window of tokens at a time, set in Ollama by num_ctx. When a prompt doesn't fit, Ollama keeps the end and silently drops the beginning, with no error. On the Larch Grove minutes, that meant the model lost the question and summarized, or kept the question, lost the name, and said NONE. prompt_eval_count is how you spot it. A big enough window fixes it at a memory cost; putting the instructions last makes failures safer; and sending only the part of the page that matters fixed it in the smallest window of all. Lesson 7 is about the other end of the request: making the model's answer come back in a fixed shape that code can rely on.

Check yourself

Key terms

• Context window: how many tokens a model works with at once, counting the system message, prompt, material and answer.
• num_ctx: Ollama's setting for the window size, sent in a request's options.
• Context length: the most a model was built to handle, shown by ollama show. num_ctx can be smaller.
• Silent cut: what happens when a prompt is bigger than the window: Ollama drops the start, with no error.
• prompt_eval_count: how many prompt tokens the model actually read. Much smaller than expected means a cut.
• Focusing: sending only the part of a document that matters, so it fits a small window.

Go deeper

• The practice set (opens in a new tab): context.py and the long page are in it. Try writing a better focus() that keeps Alder Creek's and Heron Point's names too, and check it against all ten pages.
• Ollama's API reference (opens in a new tab), for num_ctx and the other options you can set per request.
• Ollama's Context length (opens in a new tab) page: the default window for each amount of graphics memory, and how to change it for every request at once.