Talking to it from code

Why code

Typing ollama run is fine for trying things. Real work needs more: the same question asked of a hundred pages, settings you can control, and answers you can check automatically. That means code.

Here's the good news: you already know how. Since Lesson 2, every ollama run has really been the ollama command sending a request to Ollama's background service, which does the actual work. In this lesson you send those requests yourself. The Python track's Talking to APIs lesson covers the basics of requests and JSON, if you want a refresher.

The practice set

From here on, the hands-on parts use a practice set: ten web pages from Larkspur County, a county I made up, plus an answer key. Each page tests a different trap: a former mayor, an acting mayor, a mayor-elect, a council table, a town with no mayor at all, and more. Because the county is fictional, the right answers never change, and no real person is ever named by mistake.

Get it, set up a Python virtual environment, and make sure Ollama is running:

git clone https://github.com/Kwamib/local-llm-practice.git
cd local-llm-practice
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

If you stopped Ollama after Lesson 3, start it again: open the app, or run brew services run ollama. The model is the same qwen2.5:3b.

1. The API, with curl

Ollama's service listens at http://localhost:11434. "localhost" means this computer, and 11434 is the port, the numbered door the service waits behind. Ask it which models you have:

curl -s http://localhost:11434/api/tags | python3 -m json.tool | head -20

{
    "models": [
        {
            "name": "qwen2.5:3b",
            "model": "qwen2.5:3b",
            "modified_at": "2026-10-27T13:07:20.563595763-04:00",
            "size": 1929912432,
            "digest": "357c53fb659c5076de1d65ccb0b397446227b71a42be9d1603d46168015c9e4b",
            "details": {
                "parent_model": "",
                "format": "gguf",
                "family": "qwen2",
                "families": [
                    "qwen2"
                ],
                "parameter_size": "3.1B",
                "quantization_level": "Q4_K_M",
                "context_length": 32768,
                "embedding_length": 2048,
                "runner": "ggml"

That's ollama list and ollama show as data: the same 3.1B parameters and Q4_K_M quantization, ready for a program to read. (python3 -m json.tool just lays the JSON out neatly.)

Now ask a question. The endpoint for a one-off question is /api/generate, and you send it a small JSON body:

curl -s http://localhost:11434/api/generate -d '{"model": "qwen2.5:3b", "prompt": "In one sentence, what is a container in software?", "stream": false}' | python3 -m json.tool

{
    "model": "qwen2.5:3b",
    "created_at": "2026-10-27T20:35:01.664235Z",
    "response": "A container is a software package that bundles an application and all its dependencies together to ensure consistent deployment across different environments.",
    "done": true,
    "done_reason": "stop",
    "context": [
        151644,
        8948,
        198,
        2610,
        525,
        1207,
        16948,
        ...
    ],
    "total_duration": 2064578666,
    "load_duration": 1812796791,
    "prompt_eval_count": 40,
    "prompt_eval_cached_count": 0,
    "prompt_eval_duration": 66559000,
    "eval_count": 24,
    "eval_duration": 177600000
}

The context list is 63 numbers long; it's trimmed here. What the fields mean:

• response is the answer. In a script, it's usually the only field you need.
• done_reason: "stop" means the model chose to stop, by picking its end-of-answer token. If it had hit a length limit instead, you'd see "length".
• context is the whole exchange as token IDs: the numbers the model actually reads and writes, one per token. Lesson 1 said a model works in tokens; this is them. 151644 marks where a message starts. The first message isn't yours: it's the hidden system message from Lesson 2, "You are Qwen, created by Alibaba Cloud…", which is why prompt_eval_count is 40 tokens for a 9-word question.
• The durations are in nanoseconds, billionths of a second. total_duration: 2064578666 is 2.06 seconds, and load_duration shows 1.8 of those were loading the model, the same thing --verbose showed you in Lesson 2.

And "stream": false? By default, Ollama streams: it sends each token as soon as it's written, as its own small piece of JSON. That's what makes chat windows show text appearing word by word. For a script, one complete reply is simpler, so you turn streaming off. Try both settings below, along with temperature:

{
  "model": "qwen2.5:3b",
  "prompt": "Suggest a name for a new coffee shop in a small town. Reply with the name only.",
  "stream": false,
  "options": {
    "temperature": 0
  }
}

2. The same thing from Python

ask.py in the practice set does in Python what you just did with curl. It has three parts. First, a function that sends one prompt and returns the answer:

OLLAMA_URL = "http://localhost:11434/api/generate"
MODEL = "qwen2.5:3b"


def ask(prompt, temperature=0):
    payload = {
        "model": MODEL,
        "prompt": prompt,
        "stream": False,
        "options": {"temperature": temperature},
    }
    r = requests.post(OLLAMA_URL, json=payload, timeout=120)
    r.raise_for_status()
    return r.json()["response"].strip()

The body is the same JSON, with one addition: options, where model settings go. Here that's the temperature, which defaults to 0, so the same page always gets the same answer. You couldn't set that from ollama run; from code it's one line. The timeout stops the script from waiting forever if something goes wrong, and raise_for_status() turns an error reply into a Python error, so a failure can't pass for an answer.

Second, a way to turn a web page into text. The model doesn't need the HTML tags, only the words, so this uses Python's built-in HTML parser to keep the visible text:

class TextOnly(HTMLParser):
    """Collect the visible text of an HTML page, one piece per line."""

    def __init__(self):
        super().__init__()
        self.parts = []

    def handle_data(self, data):
        if data.strip():
            self.parts.append(data.strip())


def page_text(path):
    parser = TextOnly()
    with open(path, encoding="utf-8") as f:
        parser.feed(f.read())
    return "\n".join(parser.parts)

Third, the prompt. It's the same narrow question from Lesson 2, with the page dropped into the {page} slot:

PROMPT = """Read this web page and answer with only the full name of the person currently serving as mayor or acting mayor, nothing else.
If the page does not name a current mayor or acting mayor, answer NONE.

PAGE:
{page}"""

Try it on one page:

python3 ask.py pages/cedar-bend.html

Dana Whitfield

Compare that with Lesson 2, where the same page in the terminal got "Mayor Dana Whitfield", title included. This time, with the temperature at 0 and a prompt that also says what to do when there's no mayor, it answered with just the name. Small changes to settings and wording change answers; Lesson 5 is about that.

3. All ten pages, scored

Run it with no page, and ask.py goes through answers.csv, asks about every page, and compares each answer with the right one:

python3 ask.py

ok   Cedar Bend     expected Dana Whitfield   got Dana Whitfield
ok   Elm Ridge      expected Carla Mendes     got Carla Mendes
ok   Alder Creek    expected Grace Okafor     got Grace Okafor
ok   Birch Hollow   expected NONE             got NONE
ok   Fox Run        expected Lena Park        got Lena Park
ok   Juniper Flats  expected Tom Haskell      got Tom Haskell
ok   Heron Point    expected Samuel Adeyemi   got Samuel Adeyemi
ok   Quarry Hill    expected NONE             got NONE
MISS Willow Bend    expected Robert Chen      got Robert "Bob" Chen
ok   Stone Mill     expected Ana Silva        got Ana Silva

9 of 10 exactly right

Nine out of ten, from a 3B model on a laptop. It found the acting mayor, skipped the mayor-elect, picked the mayor out of a seven-row table, found a name hidden in a footer and a signature, ignored another town's mayor, and said NONE twice when there was no mayor to find.

Now look at the miss. The page says Mayor Robert "Bob" Chen, and the model answered Robert "Bob" Chen. That's the right person. The script calls it a miss because it compares the two answers exactly, letter for letter, and the quotes and nickname don't match.

So which is wrong, the model or the test? A bit of both, and there are two honest fixes:

• Ask more precisely. The prompt never said what to do with nicknames. Lesson 5 is about prompts that leave less room for guessing.
• Compare more fairly. A person reading the results would count this as right. Code can too, by cleaning up names before comparing them. Lesson 10 is about tests that are strict where it matters and fair where it doesn't.

That's the real lesson of the score. A test only means something if you know what it's measuring. "9 of 10 exactly right" is true; "9 of 10 right" would not be.

4. Temperature, from code

temperature.py asks one creative question three times at temperature 0, then three times at 1:

python3 temperature.py

temperature 0:
   Brew & Bean
   Brew & Bean
   Brew & Bean
temperature 1:
   Brew & Leaf
   Morning Brew
   Morning Brew

At 0, the same answer every time: exactly what you want when a script reads names off pages. At 1, variety: good for brainstorming, bad for anything you want to check. Lesson 1's widget showed this with made-up numbers; this is the real thing. It's also why ask.py sets the temperature itself instead of trusting the default, which isn't 0.

5. Who can reach it?

One more thing about that address. Ollama's API has no password. Anything that can reach port 11434 can use your models. By default that's only programs on your own computer, because Ollama listens on 127.0.0.1, which means "this machine only". You can check what it's listening on:

lsof -nP -iTCP:11434 -sTCP:LISTEN

COMMAND  PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ollama  1987 mayowababatola    3u  IPv4 0xeb9722e9801de45e      0t0  TCP 127.0.0.1:11434 (LISTEN)

That's from my homelab, where my CivicGrid check runs. 127.0.0.1:11434 means only that machine can use it. If you ever see *:11434, anything on your network can. Some guides tell you to open it up so other devices can use your model; if you do, put it behind a firewall, or reach it through an SSH tunnel instead, so it isn't open to everyone.

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

Connection refused. A script that worked yesterday now fails straight away with a connection error to localhost:11434. Nothing in the script changed. What's the likely cause? Ollama's background service isn't running, perhaps stopped at the end of the day, or not started after a restart. The script is fine; start the service and run it again.

A JSON error on a good reply. Someone writes their own version of ask(), but leaves out "stream": False. Every call now fails with a JSON error, even though Ollama is clearly answering. Why? With streaming on, Ollama sends one small JSON object per token, so the reply is many objects, not one, and r.json() can't read it as a single object. Turning streaming off fixes it.

Different answers on different days. A team's script sorts support emails, and the same email sometimes lands in a different category. They never set a temperature. What's going on? Ollama's default temperature isn't 0, so each run can pick a different token. Setting "options": {"temperature": 0} makes the answers repeatable, and testable.

Where you are now

Ollama is a service at localhost:11434, and everything the ollama command does is a request you can send yourself. /api/generate takes a model, a prompt, stream and options, and replies with JSON, durations in nanoseconds. From Python, that's one short function. With the temperature at 0 and a narrow prompt, qwen2.5:3b got 9 of the 10 practice pages exactly right, and its one miss was the right person written a different way. Lesson 5 is about writing prompts that leave less to chance, starting with that miss.

Check yourself

Key terms

• API: a way for programs to talk to each other. Ollama's is a set of web addresses on your own machine.
• Endpoint: one address in an API, like /api/generate or /api/tags.
• Port: the numbered door a service listens behind. Ollama's is 11434.
• localhost and 127.0.0.1: names for "this computer". A service listening there can't be reached from other machines.
• Streaming: sending an answer piece by piece as it's written, instead of all at once at the end.
• Options: model settings sent with a request, like temperature.
• Token IDs: the numbers a model actually reads and writes, one per token.
• Practice set: pages with known right answers, used to check a model's answers automatically.

Go deeper

• Ollama's API reference (opens in a new tab): every endpoint, every field in the request and the reply, and the options you can set.
• The practice set on GitHub (opens in a new tab): the ten pages, the answer key, ask.py and temperature.py.
• The Python track's Talking to APIs lesson, for requests, JSON and error handling in more depth.