Resources Local LLMs Local LLMs with Ollama

Local LLMs with Ollama · Lesson 2 of 13 · about 30 minutes

Your first local model

Install Ollama, download a small model, and run it on your own laptop. You'll read what the model tells you about itself, see how much memory it really uses, watch it invent a mayor for a town it says doesn't exist, and then see what changes when you hand it the facts.

By the end you'll have

  • Ollama installed, and a small model running on your own laptop, for free
  • Read what a model's details mean: size, parameters, context and licence
  • Seen how much memory a model really uses, and how to free it
  • Watched a model make things up, and seen what changes when you give it the facts
  • Cleaned up, so nothing keeps running when you're done

What you need

You'll use Ollama, a free tool that downloads models and runs them on your machine. It works on macOS, Windows and Linux, and it gives you two things: a command, ollama, for running models from the terminal, and a small background service that keeps the model loaded and answers requests. You'll use the command in this lesson and talk to the service from code in Lesson 4.

The model is Qwen 2.5 3B (qwen2.5:3b), a small model from Alibaba's Qwen team. It's the smaller sibling of the model my CivicGrid project uses in production, so everything you see here carries over.

You need:

  • A laptop with at least 8GB of memory. The model uses about 3GB while it's running.
  • About 3GB of free disk space: about 2GB for the model and the rest for Ollama.
  • On a Mac, macOS Sonoma (version 14) or newer.

Nothing in this lesson costs money, and nothing you type leaves your computer.

1. Install Ollama

On macOS, download the app from ollama.com/download, open Ollama.dmg, and drag Ollama into Applications. Open it once: it asks permission to add the ollama command to your terminal, and a llama icon appears in your menu bar. That icon is the background service.

If you use Homebrew, you can install it from the terminal instead, which is also easier to remove later:

BASH
brew install ollama
brew services start ollama

brew services start runs the background service now and every time you log in. If you'd rather start it only when you need it, use brew services run ollama instead.

On Windows, download and run the installer from ollama.com/download.

On Linux, Ollama's install command is curl -fsSL https://ollama.com/install.sh | sh. That runs a script straight from the internet, so the careful habit is to download it, read it, and then run it:

BASH
curl -fsSL https://ollama.com/install.sh -o install.sh
less install.sh
sh install.sh

Then check that it worked:

BASH
ollama --version
OUTPUT
ollama version is 0.40.11
If you see "could not connect to a running Ollama instance", the command is installed but the background service hasn't started yet. Wait a few seconds and try again. If it keeps happening, open the Ollama app, or run brew services start ollama.

2. Download a model

Ollama calls downloading a model pulling it, the same word Docker uses for images:

BASH
ollama pull qwen2.5:3b
OUTPUT
pulling manifest
pulling 5ee4f07cdb9b: 100% ▕████████████████████████████████████████▏ 1.9 GB
pulling 161ddde4c9cd: 100% ▕████████████████████████████████████████▏  487 B
verifying sha256 digest
writing manifest
success

The name works like a Docker image name too: qwen2.5 is the model, and 3b is the tag, picking the size. The 1.9GB file is the model's parameters; the tiny one is its settings. "Verifying sha256 digest" means Ollama checked the download wasn't corrupted.

See what you have:

BASH
ollama list
OUTPUT
NAME          ID              SIZE      MODIFIED
qwen2.5:3b    357c53fb659c    1.9 GB    Less than a second ago

3. Read the label

Every model comes with details worth reading before you use it:

BASH
ollama show qwen2.5:3b
OUTPUT
  Model
    architecture        qwen2
    parameters          3.1B
    context length      32768
    embedding length    2048
    quantization        Q4_K_M
    template source     go_template

  Capabilities
    completion
    tools

  System
    You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

  License
    Qwen RESEARCH LICENSE AGREEMENT
    Qwen RESEARCH LICENSE AGREEMENT Release Date: September 19, 2024
    ...

The lines that matter most:

  • parameters 3.1B. About 3 billion learned numbers, the "3b" in the name. Lesson 1 explained what parameters are.
  • quantization Q4_K_M. The parameters are stored at about 4 bits each, instead of the 16 they were trained with. That's how 3 billion of them fit in 1.9GB, at a small cost in accuracy. Lesson 3 explains quantization properly.
  • context length 32768. The most tokens the model can work with at once: your question, any text you give it, and its answer, together. Lesson 6 is about this limit.
  • System. A hidden instruction the model receives before everything you type. You never see it in the conversation, but it shapes every answer. Lesson 5 shows how to replace it.
  • License. This one is easy to skip and important. This model's licence is a research licence, which limits commercial use. Licences differ between models, and even between sizes of the same family: qwen2.5:7b, the bigger sibling, uses the Apache 2.0 licence, which allows commercial use. Learning on it is fine; before you build a product on any model, read its licence.

4. Ask it something

Now run it. The --verbose flag adds timing details after the answer:

BASH
ollama run qwen2.5:3b --verbose "In one sentence, what is a container in software?"
OUTPUT
A container is a software package that bundles an application and its dependencies together to deliver it to any platform.

total duration:       1.703608917s
load duration:        1.307661833s
prompt eval count:    40 token(s)
prompt eval duration: 235.554ms
prompt eval rate:     169.81 tokens/s
eval count:           22 token(s)
eval duration:        155.296ms
eval rate:            141.66 tokens/s

A decent answer, in under two seconds. Most of that time was loading the model into memory; the answer itself took a sixth of a second.

If you leave off the question, ollama run qwen2.5:3b opens a chat in your terminal instead. Type /bye to leave it.

5. See what it's using

While the model is loaded, ask Ollama what's running:

BASH
ollama ps
OUTPUT
NAME          ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
qwen2.5:3b    357c53fb659c    2.8 GB    100% GPU     32768      llamacpp    4 minutes from now

Select each line of both outputs to see what it means:

Try it: what does each line mean?

Select a line to see what it means.

Two numbers to keep in mind. 2.8GB is the real memory cost, more than the 1.9GB file, because Ollama sets aside room for the conversation too. And 141 tokens a second came from a high-end laptop chip, an Apple M4 Max. Yours may well be slower, and that's fine for everything in this track.

6. Watch it make something up

Lesson 1 said a model continues text plausibly rather than checking facts. Here's what that looks like with a real model. Maple Falls, Ohio, is the town I made up for Lesson 1. First, ask about its mayor:

BASH
ollama run qwen2.5:3b "Who is the mayor of Maple Falls, Ohio?"
OUTPUT
I'm sorry, but I don't have specific information about the current mayor of Maple Falls, Ohio, as municipal governments and their leaders can
change over time. For the most accurate and up-to-date information, I recommend checking recent news sources or official government websites
related to Maple Falls, Ohio.

That looks responsible. Models trained for chat are taught to answer "who is the current…" questions this way. But read it again: it says nothing about the town not existing. It treats Maple Falls as a real place with a real mayor it just doesn't know about.

Now ask the same model three questions in a row. The first two simply assume the town is real:

BASH
ollama run qwen2.5:3b "Maple Falls, Ohio holds a famous maple syrup festival every spring. In two sentences, describe its history."
ollama run qwen2.5:3b "Write a one-sentence local news headline announcing that the mayor of Maple Falls, Ohio, was re-elected. Use the mayor's full name."
ollama run qwen2.5:3b "Does Maple Falls, Ohio exist? Answer yes or no, then one sentence."
OUTPUT
Maple Falls, Ohio's maple syrup festival has a rich history dating back to the early 19th century when Native American tribes first taught settlers
the technique of tapping maple trees for syrup. Over the years, the festival has grown to become a beloved community event, celebrating the
region's natural resources and agricultural heritage.

Mayor Emily Mitchell Re-elected for Another Term in Maple Falls, Ohio

No

That's the whole lesson in three answers. The festival history is invented. Emily Mitchell is invented. And asked directly, the same model says the town doesn't exist. It isn't lying, and it isn't confused in the way a person would be: each answer is simply the most plausible continuation of the question it was given. A question that takes the town for granted gets an answer that does too.

It also skipped "then one sentence" in the last answer. Small models don't always follow every part of an instruction, which is one reason Lesson 5 is about writing prompts that ask for one thing.

Your answers may be worded differently. Ollama's default temperature isn't 0, so each run can pick different words. The pattern is what to look for.

7. Hand it the facts

Now do what Lesson 1 recommended: give the model the text and ask a narrow question about it. Cedar Bend is another made-up village, and this time the model gets its web page:

BASH
ollama run qwen2.5:3b "Read this page and answer with only the current mayor's full name, nothing else. PAGE: Welcome to the Village of Cedar Bend. Village Council: Mayor Dana Whitfield; Council members Luis Ortega, Priya Raman and Tom Becker. Mayor Whitfield was elected in 2025, succeeding former mayor Harold Greene, who served from 2013 to 2025. The council meets the first Tuesday of each month at Village Hall."
OUTPUT
Mayor Dana Whitfield

Run it again, and you get the same answer. It picked the right person out of five names on the page, including a former mayor. Now ask about something the page doesn't say:

BASH
ollama run qwen2.5:3b "Read this page and answer with only the village clerk's full name. If the page does not name a clerk, answer NONE. PAGE: Welcome to the Village of Cedar Bend. Village Council: Mayor Dana Whitfield; Council members Luis Ortega, Priya Raman and Tom Becker. Mayor Whitfield was elected in 2025, succeeding former mayor Harold Greene, who served from 2013 to 2025. The council meets the first Tuesday of each month at Village Hall."
OUTPUT
NONE

Same model, completely different behaviour. Given the facts and a way to say "not here", it read the page instead of inventing. That's the pattern the rest of this track builds on.

Notice one small thing, though. You asked for "only the full name", and it answered "Mayor Dana Whitfield", title included. Close, but not exactly what you asked for, and if a program were comparing that answer to "Dana Whitfield", it wouldn't match. Small slips like this are normal, and code that uses a model's answers has to expect them.

8. Stop it and clean up

The model unloads itself after a few minutes of not being used, but you can free the memory straight away:

BASH
ollama stop qwen2.5:3b
ollama ps
OUTPUT
NAME    ID    SIZE    PROCESSOR    CONTEXT    RUNNER    UNTIL

An empty list means nothing is loaded and no memory is being used. The model file is still on disk, ready for next time. Keep it if you're going on to the next lesson: Lessons 3 to 9 all use it.

When you're done with the track, remove the model to get the 1.9GB back:

BASH
ollama rm qwen2.5:3b

To stop the background service, quit Ollama from the menu bar (or the system tray on Windows), or with Homebrew:

BASH
brew services stop ollama

To remove Ollama completely, with Homebrew it's brew uninstall ollama, then brew autoremove to remove the libraries it installed alongside it, then rm -rf ~/.ollama to delete any models left behind. If you installed the app, Ollama's macOS page lists every folder it uses. One catch if you copy those commands: a few of the paths there are wrapped in quotes, like "~/Library/Application Support/Ollama", and the shell doesn't expand ~ inside quotes, so those lines quietly delete nothing. Write them as ~/Library/Application\ Support/Ollama instead.

In my real setup

CivicGrid's weekly mayor check uses the bigger qwen2.5:7b on my homelab, and it runs into the "Mayor Dana Whitfield" problem from step 7. A model asked for a name sometimes adds a title, so before comparing anything, the script strips titles like "Mayor" from the name, and it checks that the name really appears on the page. Getting the facts in front of the model makes it far more reliable, but not perfect.

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

Installed, but not found. Someone installs the Ollama app on a Mac, opens Terminal, types ollama --version, and gets command not found: ollama. What happened? The app adds the ollama command to the terminal the first time it's opened, and asks permission to do it. Either they never opened the app, or they declined that request. Opening Ollama and approving it fixes it.

Slow, then fast. Someone asks the model a question, and it takes several seconds to start answering. They ask another straight away, and it starts almost instantly. Is something wrong? No. The first question included the load duration: reading the model from disk into memory. Once it's loaded, it stays loaded for a few minutes, so the next questions skip that step.

Bigger is better? Someone with a 16GB laptop reads that bigger models are smarter, pulls a 70-billion-parameter model, and finds it either refuses to load or answers one word every few seconds. What went wrong? A model that size needs far more memory than the laptop has, so it either can't load or has to run mostly on the slower processor. Matching a model to your machine is what Lesson 3 is about.

Where you are now

You've installed Ollama, pulled a model, read its label, and run it. You know a loaded model costs memory and an idle one doesn't, and how to stop and remove it. And you've seen both sides of a real model: the same one that says Maple Falls doesn't exist will invent a festival history and a mayor when a question assumes the town is real, while given a page and a way to say "not here", it reads carefully and says NONE. Lesson 3 is about choosing a model that fits your machine: what sizes and quantization really mean.

Check yourself

  1. What does ollama pull qwen2.5:3b do?

  2. The model file is 1.9GB, but ollama ps shows 2.8GB. Why?

  3. Your first question takes seconds; the next is almost instant. Why?

  4. The model says Maple Falls doesn't exist, but names its mayor. How?

  5. Why did asking about the Cedar Bend page give reliable answers?

  6. What does ollama stop qwen2.5:3b do?

Key terms

  • Ollama: a free tool for downloading and running models on your own machine.
  • Background service: the part of Ollama that keeps models loaded and answers requests. The ollama command talks to it.
  • Pull: download a model to your machine.
  • Tag: the part of a model's name after the colon, usually its size, like 3b or 7b.
  • Quantization: storing a model's parameters with fewer bits, so it's smaller and faster, at a small cost in accuracy.
  • Context length: the most tokens a model can work with at once, counting your input and its answer.
  • System message: a hidden instruction the model receives before everything you type.
  • Load duration: the time to read a model from disk into memory, paid only when it isn't already loaded.
  • Eval rate: how fast a model writes, in tokens per second.

Go deeper

From www.kwamib.dev/resources/local-llms/your-first-local-model