By the end you'll know
- The parts of a prompt that does one job, and why each one is there
- What a system message is, and how to send your own
- Why adding instructions can make a small model worse, not better
- Why a wrong answer is worse than no answer
- How to test a prompt change, and when to fix things in code instead
Where you are
In Lesson 4, ask.py scored 9 out of 10 on the practice pages. Its one miss was Willow Bend: the page says Mayor Robert "Bob" Chen, and the model answered Robert "Bob" Chen when the answer key says Robert Chen.
The obvious fix is to tell the model what you want: no nicknames. That's what most people would do, and what most prompt guides suggest: be specific, add rules, give examples. This lesson does exactly that, carefully, and measures every step. The results surprised me, and they're the most useful thing in this track so far.
A prompt that does one job
Here's the prompt that scored 9 out of 10. It's short, and every line has a purpose:
Read this web page and answer with only the full name of the person currently serving as mayor or acting mayor, nothing else.
If the page does not name a current mayor or acting mayor, answer NONE.
PAGE:
{page}
- One task. Find one person. Not "summarize the page" or "tell me about the council".
- A fixed answer shape. "Only the full name, nothing else" means the answer can be compared with something, by code, without a person reading it.
- An honest way out. "If the page does not name… answer NONE." Without it, a model that can't find a mayor has to say something, and you've seen what models do then. With it, "I don't know" has a name.
- The input, clearly marked.
PAGE:and then the page, at the end, so the instructions and the material don't run together.
Seven prompts, ten pages
prompts.py in the practice set runs several prompts against all ten pages, at temperature 0, and scores each one exactly the way ask.py does. Each prompt is a name, an optional system message (more on that below), and the prompt text:
PROMPTS = {
"vague": {
"system": None,
"prompt": "Who is the mayor?\n\n{page}",
},
"narrow": {
"system": None,
"prompt": """Read this web page and answer with only the full name ...""",
},
...
}
Run all of them, or name the ones you want:
python3 prompts.py
The first round tried four. Here's what came back:
== vague: 0 of 10 exactly right
MISS Cedar Bend expected Dana Whitfield got Based on the information provided in the text, the current mayor of the fictional Village of Cedar Bend is Dan...
MISS Elm Ridge expected Carla Mendes got Based on the information provided, the current acting mayor of the fictional Town of Elm Ridge is Mayor Pro Te...
MISS Alder Creek expected Grace Okafor got Based on the information provided, the mayor of the fictional city of Alder Creek is Grace Okafor. The passage...
MISS Birch Hollow expected NONE got Based on the information provided, the Town of Birch Hollow does not have a mayor. Instead, it has a town mana...
MISS Fox Run expected Lena Park got Based on the information provided in the text, the mayor of the fictional town of Fox Run is Lena Park. The te...
MISS Juniper Flats expected Tom Haskell got Based on the information provided in the election results for Juniper Flats, the mayor elected in the November...
MISS Heron Point expected Samuel Adeyemi got You've provided a message from a fictional mayor, Samuel Adeyemi, for a fictional town called Heron Point. The...
MISS Quarry Hill expected NONE got Based on the information provided in the text, the mayor of Quarry Hill does not have a specific name mentione...
MISS Willow Bend expected Robert Chen got Based on the information provided in the text, the mayor of Willow Bend is Mayor Robert "Bob" Chen. Here are s...
MISS Stone Mill expected Ana Silva got Based on the information provided in the text, the mayor of Stone Mill is Ana Silva. The text states that "Sto...
== narrow: 9 of 10 exactly right
MISS Willow Bend expected Robert Chen got Robert "Bob" Chen
== precise: 5 of 10 exactly right
MISS Elm Ridge expected Carla Mendes got NONE
MISS Fox Run expected Lena Park got NONE
MISS Juniper Flats expected Tom Haskell got NONE
MISS Heron Point expected Samuel Adeyemi got NONE
MISS Willow Bend expected Robert Chen got NONE
== precise+system: 7 of 10 exactly right
MISS Fox Run expected Lena Park got NONE
MISS Juniper Flats expected Tom Haskell got NONE
MISS Heron Point expected Samuel Adeyemi got NONE
Long answers are cut at 110 characters by the script; the real ones went on much longer.
vague scored 0, and that's not quite fair to it. Read the answers: most of them name the right person. A human reading them would be satisfied. But every one is a paragraph starting "Based on the information provided", which no program can compare with anything. A prompt for code has to fix the answer's shape, not just its topic.
precise is the one to stare at. It's narrow with one extra line, written to fix Willow Bend:
Write the name as first name and last name only: no titles, no nicknames, no quotation marks.
It didn't fix Willow Bend. Worse, four pages the model had right a moment ago, Elm Ridge, Fox Run, Juniper Flats and Heron Point, now came back NONE. One reasonable-sounding line took the score from 9 to 5.
Why? I can't see inside the model, so this is a guess. Every page it gave up on has the name next to a title, or in an odd spot: "Mayor Pro Tem Carla Mendes", a footer, a signature, "Mayor Tom Haskell continues to serve". With a list of things the answer mustn't contain, a small model seems to have taken the safe way out, NONE, rather than work out how to strip them. What isn't a guess is the general lesson: you can't predict what a prompt change will do. You have to measure it, on every input.
The system message
Lesson 2 showed that every model comes with a hidden system message, an instruction it receives before anything you type. For qwen2.5:3b it's "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." You can replace it. In Ollama's API, it's one more field in the request:
def generate(prompt, system=None):
payload = {
"model": MODEL,
"prompt": prompt,
"stream": False,
"options": {"temperature": 0},
}
if system:
payload["system"] = system
r = requests.post(OLLAMA_URL, json=payload, timeout=120)
r.raise_for_status()
return r.json()["response"].strip()
A system message is the place for standing rules, the ones that apply to every question: who the model is working for, and what it must never do. The one in prompts.py is:
You read web pages for a data team. Answer only from the page you are given. Never use outside knowledge. If the page does not contain the answer, say NONE.
Added to precise, it won back two pages, from 5 to 7. So the second round tried it on narrow, the best prompt, along with two other common pieces of advice: describe what to do instead of what not to do (clearer), and show a worked example (example).
python3 prompts.py narrow+system clearer example
== narrow+system: 7 of 10 exactly right
MISS Fox Run expected Lena Park got NONE
MISS Juniper Flats expected Tom Haskell got Rosa Delgado
MISS Heron Point expected Samuel Adeyemi got NONE
== clearer: 7 of 10 exactly right
MISS Juniper Flats expected Tom Haskell got Haskell
MISS Heron Point expected Samuel Adeyemi got NONE
MISS Willow Bend expected Robert Chen got NONE
== example: 7 of 10 exactly right
MISS Fox Run expected Lena Park got NONE
MISS Juniper Flats expected Tom Haskell got NONE
MISS Willow Bend expected Robert Chen got NONE
All three scored 7. None beat narrow. The system message that helped precise made narrow worse. The worked example, which showed exactly how to handle a nickname, still didn't get Willow Bend right. Pick any prompt below to see its text and every result:
Not all misses are equal
Look closely at narrow+system's Juniper Flats answer: Rosa Delgado. That's the mayor-elect, who hasn't taken office. Among the short answers, it's the only one that named the wrong person. (The vague answer for Juniper Flats was cut off just as it mentioned "the mayor elected in the November…", so it may have made the same mistake inside its paragraph.)
Compare the three kinds of miss in these runs:
- NONE when there is a mayor. Annoying, but safe. A script can send every NONE to a person to check, and nothing false gets written down.
- The right person, written differently, like
Robert "Bob" ChenorHaskell. Code can often fix these, or a fairer comparison can accept them. - The wrong person. Dangerous. It looks exactly like a right answer, so nothing flags it. If a script trusted it, the wrong name would go into your data, and stay there.
So narrow+system is worse than its 7 out of 10 suggests. Counting right answers isn't enough; you have to look at what kind of wrong the misses are. Lesson 8 is about catching the dangerous kind.
What to take from this
- Start simple. A prompt with one task, a fixed answer shape and a way out (NONE) is hard to beat. Here, nothing beat it.
- Test every change on every input. A fix for one page broke four others. You'd never see that by trying the page you were fixing.
- Count the kinds of mistakes, not just the score. One wrong person matters more than several NONEs.
- Small models are touchy. A 3B model can flip from right to NONE because of one extra sentence. Bigger models are usually steadier, and as Lesson 3 showed, cost more to run. Either way, the test tells you, not the size.
- Some fixes belong in code. The nickname beat four prompts. Removing a nickname in quotes from an answer is a couple of lines of code, and it works every time. Lesson 10 does exactly that, along with fairer scoring.
The prompt in ask.py stays as it is: narrow, 9 out of 10.
In my real setup
CivicGrid's weekly mayor check uses a much longer prompt than narrow, on the bigger qwen2.5:7b. It lists the equivalent offices for towns without a mayor, such as a village president or a select board chair, and has rules like "Do not return a former or past officeholder" and "Do not use any knowledge from outside this page".
On nicknames, it made the same choice this lesson ended with. The prompt says the name "MUST appear verbatim in the page text below. Copy it exactly." So the model doesn't try to remove nicknames at all; it copies the name as the page writes it. Then code works out that a name like "Bob Chen" and one like Robert "Bob" Chen are the same person. That's more reliable than asking the model to do it, for exactly the reasons you just saw.
Whether every one of those rules earns its place, I don't know. After this lesson, I'm suspicious of long prompts on small models. Lesson 10 is how to find out.
Spot the problem
Three situations. Work out what's wrong before you read the explanation.
The fix that broke things. A team's extraction prompt misses one tricky document, so they add a sentence that handles it. They test it on that document, it works, and they ship it. A week later, someone notices the results are much worse overall. What went wrong? They tested the change only on the input it was meant to fix. As precise showed, one added line can break inputs that used to work. Every prompt change needs the whole test set.
A great score with one bad answer. A prompt scores 9 out of 10, so it goes into production. The one miss was a confident wrong name, not a NONE. Is 9 out of 10 good enough? Not without looking at the miss. A wrong answer is trusted and saved; a NONE gets checked by a person. A prompt with 8 right and 2 NONEs can be safer than one with 9 right and 1 wrong.
Polite paragraphs. A script asks a model "Who is the mayor?" about thousands of pages, and the database fills up with entries like "Based on the information provided, the mayor is…". What was missing? An answer shape. The model answered a person's question, politely. A prompt for code has to say exactly what to return, like "only the full name, nothing else".
Where you are now
A prompt that does one job has one task, a fixed answer shape, an honest way out, and its input clearly marked. A system message holds standing rules, and you can send your own. But on a small model, more instructions can make things worse: of seven prompts, the simple one from Lesson 4 still scored best, at 9 out of 10, and a well-meant system message led the model to name the mayor-elect: the wrong person. So test every change on every page, count what kind of mistakes you get, and move fixes into code when the prompt can't make them reliable. Lesson 6 looks at the other half of every prompt: the page itself, and what happens when it's too long for the model to take in.
Check yourself
What does "answer NONE" in a prompt give the model?
Why did the vague prompt score 0, even with mostly right names?
One added line took a prompt from 9 right to 5. What's the lesson?
What is a system message for?
Which miss is the most dangerous?
Four prompts failed to remove a nickname. What's the better fix?
Key terms
- Prompt: the text you send a model: instructions, plus the material to work on.
- Answer shape: what the answer must look like, like "only the full name". It's what lets code use the answer.
- Way out: an allowed answer for "not here", like NONE, so the model doesn't have to invent one.
- System message: an instruction the model receives before every prompt, for standing rules. Each model has a default; you can send your own.
- Worked example: an example of an input and the answer you want, included in the prompt.
- Safe miss and dangerous miss: a NONE that a person will check, versus a confident wrong answer that nothing flags.
Go deeper
- The practice set (opens in a new tab):
prompts.pyhas all seven prompts. Add your own to thePROMPTSlist and run it by name. - Ollama's API reference (opens in a new tab), for the
systemfield and the other request fields. - Prompt engineering overview (opens in a new tab), Anthropic's guide to writing clear prompts. It's written for much bigger hosted models, so read it alongside this lesson's results: advice that works on a large model can still need testing on a small one.