Picking a model your machine can run

Where you are

In Lesson 2 you ran qwen2.5:3b and read its label: 3.1 billion parameters, quantization Q4_K_M, 1.9GB on disk, 2.8GB in memory. Open the Ollama model library (opens in a new tab) and you'll find the same model in many more versions: 0.5b, 1.5b, 3b, 7b, 14b, 32b, 72b, and each of those in several quantizations. Which one should you run?

The usual advice is "the biggest one your machine can handle". This lesson tests that advice on a real laptop, and it doesn't hold up as well as you'd expect.

This is one of the tougher lessons in the track, because two ideas meet here: size and precision. Take it slowly; the measurements do most of the explaining.

Size: how many numbers

Lesson 1 described parameters as the learned numbers that make up a model. The size in the name is how many there are: 3b is about 3 billion, 7b about 7 billion.

More parameters means more room to store patterns, so bigger models generally write better, follow instructions more reliably and know more. But every one of those numbers has to sit in memory while the model runs, and every word it writes means working through all of them. So size costs you three things: disk space, memory and speed.

Quantization: how precisely each number is stored

A model is trained with each number stored in 16 bits. Quantization stores them with fewer bits, rounding each one, so the whole model gets smaller and faster at a small cost in accuracy:

• 16-bit is the original precision. Biggest, slowest, and what the model was trained with.
• 8-bit (shown as q8_0) is about half the size, and very close to the original.
• 4-bit (shown as q4_K_M, or Q4_K_M in ollama show) is about a quarter of the size. This is Ollama's default for most models, because for most jobs the loss is hard to notice.

When you pull qwen2.5:3b with no other detail, you get the 4-bit version. To get another one, you name it in the tag, like qwen2.5:3b-instruct-q8_0.

Three versions, measured

Here's the test. Three versions of the same model on one laptop: the 3b you already know, the same 3b at 8-bit, and the 7b at 4-bit. Each pulled, then given the same page with a trap in it:

ollama pull qwen2.5:3b-instruct-q8_0
ollama pull qwen2.5:7b
ollama list

NAME                        ID              SIZE      MODIFIED
qwen2.5:7b                  845dbda0ea48    4.7 GB    Less than a second ago
qwen2.5:3b-instruct-q8_0    cd6aa8b25d7a    3.3 GB    About a minute ago
qwen2.5:3b                  357c53fb659c    1.9 GB    53 minutes ago

The page is about a town whose mayor resigned. The council's Mayor Pro Tem is now acting mayor, and the old mayor's name is still on the page. A model that grabs the first name next to the word "Mayor" gets it wrong. This loop asks each version the same question, prints its answer, its speed and its memory, and unloads it before the next:

PAGE="Town of Elm Ridge. Town Council: Mayor Pro Tem Carla Mendes; Council members Brian Oduya and Steve Kowalski. Mayor Raymond Ellis, who had served since 2017, resigned in August 2026. Mayor Pro Tem Mendes is serving as acting mayor until a special election in March 2027."
for m in qwen2.5:3b qwen2.5:3b-instruct-q8_0 qwen2.5:7b; do echo "== $m"; ollama run $m --verbose "Read this page and answer with only the full name of the person currently leading the town as mayor or acting mayor, nothing else. PAGE: $PAGE" 2>&1 | grep -v "duration\|prompt eval\|eval count"; ollama ps | tail -1; ollama stop $m; done

== qwen2.5:3b
Carla Mendes

eval rate:            136.56 tokens/s

qwen2.5:3b    357c53fb659c    2.8 GB    100% GPU     32768      llamacpp    4 minutes from now
== qwen2.5:3b-instruct-q8_0
Carla Mendes

eval rate:            113.48 tokens/s

qwen2.5:3b-instruct-q8_0    cd6aa8b25d7a    4.2 GB    100% GPU     32768      llamacpp    4 minutes from now
== qwen2.5:7b
Carla Mendes

eval rate:            102.57 tokens/s

qwen2.5:7b    845dbda0ea48    5.7 GB    100% GPU     32768      llamacpp    4 minutes from now

Put side by side:

All three got it right. The 8-bit version used 1.5 times the memory of the 4-bit one and was slower, for the same answer. The 7b used twice the memory and was slower again, for the same answer. For this job, the smallest version was enough.

Two notes on the numbers. Memory in use is always about a gigabyte more than the file, because Ollama sets aside room for the conversation; Lesson 6 explains that room. And the speed gap is unusually small here. These came from a high-end laptop chip, an Apple M4 Max, which moves data around unusually fast. On an ordinary laptop, a 7b model is usually much slower than a 3b, not just 25% slower.

Bigger isn't more honest

Accuracy on a page is one thing. What about the Maple Falls test from Lesson 2, where the model is asked about a town that doesn't exist? Here's the 7b, asked the same questions the 3b was asked:

ollama run qwen2.5:7b "Write a one-sentence local news headline announcing that the mayor of Maple Falls, Ohio, was re-elected. Use the mayor's full name."
ollama run qwen2.5:7b "Write a one-sentence local news headline announcing that the mayor of Maple Falls, Ohio, was re-elected. Use the mayor's full name."
ollama run qwen2.5:7b "Does Maple Falls, Ohio exist? Answer yes or no, then one sentence."

Mayor Johnathan Maple Secures Another Term, Re-elected as Mayor of Maple Falls, Ohio.

Mayor Jameson Thompson Secures Another Term as Leader of Maple Falls, Ohio in Recent Election.

Yes, Maple Falls, Ohio exists.

Two different invented mayors in two runs. And asked directly, the 7b said the town exists, where the 3b in Lesson 2 said it didn't. One run each isn't proof that the 3b is more careful; ask again and either might flip. What it does show is that a bigger model is just as willing to invent, and sounds every bit as sure. Size buys you better writing and better instruction-following. It doesn't buy you a model that knows what it doesn't know.

That's why the rest of this track doesn't solve made-up answers with bigger models. It solves them with what you saw at the end of Lesson 2: give the model the facts, ask a narrow question, and check the answer.

Will it fit?

The question that matters first is simpler: will the model fit in your machine's memory, with room left for everything else? Ollama's own guidance is that you need at least 8GB of memory for 7B models, 16GB for 13B models and 32GB for 33B models. Pick your machine's memory below to see how the three measured versions, and two bigger ones, compare:

If a model is too big for the graphics chip, Ollama usually doesn't refuse it. It runs part of the model on the processor (CPU) instead of the graphics chip (GPU), and you'll see it in ollama ps as a split, like 40%/60% CPU/GPU instead of 100% GPU. It still works, but it can be many times slower. 100% GPU in ollama ps is the sign a model really fits. On a machine without a suitable graphics chip, everything runs on the processor, which works for small models and gets slow fast for bigger ones.

How to choose

Put it together, and picking a model is a short checklist:

1. Start small. Begin with a 3b or similar at the default 4-bit. It runs on almost anything.
2. Write a test. A few inputs where you know the right answer, like the Elm Ridge page. Include at least one with a trap.
3. Move up only when the test fails. If the small model gets your test wrong, try the next size. If it passes, a bigger model mostly costs you memory and speed.
4. Check it fits. After running it, look for 100% GPU in ollama ps.
5. Read the licence. In Lesson 2 you saw that qwen2.5:3b has a research licence, while qwen2.5:7b uses Apache 2.0, which allows commercial use. If you're building something you'll sell, that alone can decide it.

Step 2 is the important one, and it's the subject of Lesson 10. A model you've tested beats a bigger model you've only read about.

When you're done comparing, remove the versions you don't need. Each one is gigabytes of disk:

ollama rm qwen2.5:3b-instruct-q8_0
ollama rm qwen2.5:7b
ollama list

deleted 'qwen2.5:3b-instruct-q8_0'
deleted 'qwen2.5:7b'
NAME          ID              SIZE      MODIFIED
qwen2.5:3b    357c53fb659c    1.9 GB    54 minutes ago

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

Could not connect. Someone finishes for the day by stopping Ollama's background service. The next morning they run ollama run qwen2.5:3b "hello" and get Error: could not connect to ollama server, run 'ollama serve' to start it. Is the model broken? No. The ollama command is only a client: it sends your prompt to the background service, which does the work. With the service stopped, there's nothing to send it to. Start it again (open the app, or brew services run ollama) and wait a few seconds.

It runs, but crawls. Someone with a 16GB laptop runs a 32b model. It loads, but writes about one word a second, and ollama ps shows 55%/45% CPU/GPU. What's happening? The model needs more memory than the graphics chip can use, so over half of it is running on the processor. It works, technically. A smaller model that fits completely would be many times faster, and quite possibly just as good for their job.

Upgrading for accuracy. A team switches from a 4-bit model to the 8-bit version "to be more accurate". Memory use goes up by half, answers get slower, and nobody can point to an answer that improved. What did they skip? A test. Without a set of inputs with known right answers, there's no way to tell whether the change helped. In this lesson's test, the 8-bit version gave the same answer as the 4-bit one.

Where you are now

A model's size is how many parameters it has, and quantization is how precisely each one is stored. Both trade memory and speed for quality. Measured on one laptop, the 3b at 4-bit used 2.8GB and answered the trap correctly; the 8-bit version and the 7b used up to twice the memory and answered the same. The 7b invented mayors just as readily, and even said the made-up town exists. So the way to choose is: start small, test with known answers, move up only when the test fails, and check for 100% GPU. Lesson 4 takes the model out of the terminal: you'll talk to Ollama from Python, with settings like temperature under your control.

Check yourself

Key terms

• Parameters: the learned numbers that make up a model. The size in its name, like 3b, is how many.
• Quantization: storing each parameter with fewer bits, so the model is smaller and faster at a small cost in accuracy.
• 4-bit and 8-bit: common quantization levels, shown as q4_K_M and q8_0. Ollama's default is usually 4-bit.
• GPU: the graphics chip. Models run fastest when they fit entirely on it.
• CPU/GPU split: what ollama ps shows when a model doesn't fit, and part of it runs on the slower processor.
• Test set: a few inputs with known right answers, used to decide whether a model is good enough.

Go deeper

• The Ollama model library (opens in a new tab): every model, its sizes, and the quantizations each comes in. Open a model's page and look at its tags.
• Ollama's README on GitHub (opens in a new tab), which has the memory guidance quoted in this lesson and a table of popular models with their sizes.
• The Qwen2.5 Technical Report (opens in a new tab), for how one team builds the same model at seven different sizes, and how they compare.