By the end you'll know
- What changes when a job runs with nobody watching
- How to pin a model, so a silent update can't change your results
- What a gate, a circuit breaker and a lock each protect against
- Why exit codes and a log matter more than anything printed to a screen
- Why a job that runs again and again has to remember what it already asked
Where you are
Every run in this track so far has had you watching. You started it, read the output, and noticed when something looked wrong. A scheduled job has none of that. It runs at 3 a.m. on a Sunday, and the first you hear of it is whatever it left behind.
So the question changes. It's no longer "does this work?" but "what does it do when something is wrong, and how will I find out?" A job that runs by itself has to check its own conditions before it starts, stop cleanly when they're not right, and leave a clear record either way.
This lesson builds that job from pieces you already have: Lesson 9's visible-text extraction, Lesson 8's on-page check, Lesson 10's fair names and must-pass cases. Then it breaks the job on purpose, one way at a time.
Ollama is already a service
A service is a program that runs in the background, waiting for requests, rather than one you start and watch. Ollama has been one since Lesson 2: brew services run ollama started it in the background, and every script since has talked to it at localhost:11434.
One detail matters for scheduled jobs. Ollama doesn't keep a model in memory forever. After a run, ollama ps shows how long it will stay:
ollama ps
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
qwen2.5:3b 357c53fb659c 2.3 GB 100% GPU 8192 llamacpp 4 minutes from now
By default, a model is unloaded about five minutes after its last request, which frees its 2.3GB. That's good for a laptop. For a job, it means the first question of every run waits while the model loads again, which is one reason the job below has timeouts and retries.
The job
The practice set has a new script, job.py, and a pretend database, current.csv, listing who we currently think leads each town. Two entries are out of date: Cedar Bend still lists former mayor Harold Greene, and Elm Ridge lists Raymond Ellis, who resigned.
Every run goes through six steps, in order, and any step can stop it:
- Lock. If another run is still going, skip this one.
- Preflight. Is Ollama answering? Is the model pulled? Is it exactly the model that was tested?
- Gate. Run Lesson 10's three must-pass cases. If any is wrong, refuse to run.
- Work. Check each town, using visible text only, the on-page check, and fair name comparison.
- Circuit breaker. If too many towns change, stop and write nothing.
- Proposals. Write any change to a file for a person to review. Never change the database itself.
Each way of stopping has its own exit code, the number a program hands back when it ends. 0 means it finished. Anything else means it didn't, and the number says why: 2 for preflight, 3 for the gate, 4 for the breaker. A scheduler can't read your screen, but it can read an exit code.
Pinning the model
Lesson 10 recorded each model's digest, the fingerprint of the exact file, because a name like qwen2.5:3b can point to a different file after an update. A job takes that one step further: it refuses to run with any model except the one you tested. That's pinning.
The first run has nothing pinned, so it refuses:
python3 job.py run
python3 job.py pin
python3 job.py run
2026-10-28 18:46:35 run started: qwen2.5:3b, breaker 30%
2026-10-28 18:46:35 run finished: exit 2, 0s. preflight: no pinned digest; run `python3 job.py pin` first
pinned qwen2.5:3b at 357c53fb659c in job.json
Then, with the model pinned, a normal run:
2026-10-28 19:02:24 run started: qwen2.5:3b, breaker 30%
2026-10-28 19:02:24 preflight ok: Ollama 0.40.11, qwen2.5:3b 357c53fb659c
2026-10-28 19:02:25 gate ok: every must-pass case is right
2026-10-28 19:02:26 Cedar Bend Dana Whitfield proposal
2026-10-28 19:02:26 Elm Ridge Carla Mendes proposal
2026-10-28 19:02:26 Alder Creek Grace Okafor same
2026-10-28 19:02:26 Birch Hollow NONE same
2026-10-28 19:02:26 Fox Run Lena Park same
2026-10-28 19:02:26 Juniper Flats Tom Haskell same
2026-10-28 19:02:26 Heron Point Samuel Adeyemi same
2026-10-28 19:02:26 Quarry Hill NONE same
2026-10-28 19:02:26 Willow Bend Robert "Bob" Chen same
2026-10-28 19:02:27 Stone Mill Ana Silva same
2026-10-28 19:02:27 run finished: exit 0, 3s. 2 new proposals in proposals/pending.csv, 2 for a person to see
Three seconds, and the two stale entries are found. Look at what it didn't do: current.csv still says Harold Greene. The job wrote two proposals and stopped there:
town,now,proposed,source,proposed at
Cedar Bend,Harold Greene,Dana Whitfield,pages/cedar-bend.html,2026-10-28T19:02:26
Elm Ridge,Raymond Ellis,Carla Mendes,pages/elm-ridge.html,2026-10-28T19:02:26
Every lesson since Lesson 8 has ended in the same place: the model proposes, a person decides. A job that runs by itself is where that rule matters most, because nobody is watching when it's wrong.
Breaking it on purpose
A protection you've never seen work is a guess. So each one gets tested by causing the thing it protects against.
The model changed under its name. Edit the pinned digest to something else, which is what an update looks like from the job's side:
cp job.json job.json.bak
sed -i '' 's/"digest": "[^"]*"/"digest": "000000000000"/' job.json
python3 job.py run
mv job.json.bak job.json
2026-10-28 18:46:59 run started: qwen2.5:3b, breaker 30%
2026-10-28 18:46:59 run finished: exit 2, 0s. preflight: qwen2.5:3b is now 357c53fb659c, pinned 000000000000. The model changed: run the test suite, then pin again
It stopped before asking a single question, and the message says what to do next. Lesson 10 found that swapping 3B for 7B turned a passing must-pass case into a dangerous wrong answer. A silent model update is the same kind of change, with nobody deciding to make it.
Ollama is down:
brew services stop ollama
python3 job.py run
echo "exit code $?"
python3 job.py status
brew services run ollama
2026-10-28 18:47:12 run started: qwen2.5:3b, breaker 30%
2026-10-28 18:47:12 run finished: exit 2, 0s. preflight: Ollama is not answering on localhost:11434
exit code 2
last run finished 0.0 hours ago, exit 2: preflight: Ollama is not answering on localhost:11434
A clear sentence, an exit code, and a record that status can read later. Compare that with what a script without a preflight does: a long Python error, halfway through, which nobody sees.
Too many changes. Real mayors change rarely: a few towns a year, out of many. If half the towns change in one run, it's far more likely that a website changed its layout, or the model broke, than that half the mayors left office. A circuit breaker stops the run when changes pass a limit. Here the limit is set to 10% for the test:
python3 job.py run --breaker 10
2026-10-28 18:46:48 run started: qwen2.5:3b, breaker 10.0%
2026-10-28 18:46:48 preflight ok: Ollama 0.40.11, qwen2.5:3b 357c53fb659c
2026-10-28 18:46:49 gate ok: every must-pass case is right
2026-10-28 18:46:49 Cedar Bend Dana Whitfield proposal
2026-10-28 18:46:49 Elm Ridge Carla Mendes proposal
2026-10-28 18:46:49 run finished: exit 4, 0s. circuit breaker: 2 changes after 2 of 10 towns, more than 10.0%. Real turnover is a trickle; this looks like a broken page or a broken model. Nothing written
These two changes are real, so this is a false alarm, which is the point of testing it with a low limit: you see it trip. The limit is a judgement. Set it too low and real changes keep stopping the job; too high and a broken run writes a pile of nonsense before anyone notices. Either way, a tripped breaker costs a person a look. A missing one can cost a database.
A run is already going. A slow run, a stuck run, or a person running it by hand while the schedule fires: two runs at once would ask every question twice and write the same proposals twice. A lock is a marker the first run creates and removes when it ends. Here a folder stands in for a running job:
mkdir job.lock
python3 job.py run
rmdir job.lock
2026-10-28 18:47:23 skipped: another run is still going (job.lock exists)
A skip isn't a failure, so it exits with 0: nothing is wrong, there's just nothing to do.
The gate, step 3, didn't get a test of its own here, because the pinned 3B model passes all three must-pass cases. Lesson 10 showed what would make it fail: 7B answers Rosa Delgado for Juniper Flats, so pinning 7B and running the job should stop at the gate with exit 3. That's a good exercise, if you have the space for 7B.
Try each run:
On a schedule
On macOS and Linux, the classic scheduler is cron: you give it a timetable and a command, and it runs the command at those times. This line runs the job every two minutes:
(crontab -l 2>/dev/null; echo "*/2 * * * * cd $HOME/Apps/local-llm-practice && .venv/bin/python job.py run >> logs/cron.log 2>&1") | crontab -
The five fields at the start are minute, hour, day of the month, month and day of the week; */2 in the minute field means every two minutes. Two details matter. Cron doesn't start in your project folder or with your virtual environment active, so the line changes folder first and names the virtual environment's Python directly. And >> logs/cron.log 2>&1 saves everything the job prints, errors included, because there's no screen for it to print to.
The Python track's Running it on a schedule lesson covers cron in more depth.
Left alone for five minutes, cron ran the job twice:
2026-10-28 18:50:01 run started: qwen2.5:3b, breaker 30%
2026-10-28 18:50:01 preflight ok: Ollama 0.40.11, qwen2.5:3b 357c53fb659c
2026-10-28 18:50:02 gate ok: every must-pass case is right
2026-10-28 18:50:02 Cedar Bend Dana Whitfield proposal
2026-10-28 18:50:02 Elm Ridge Carla Mendes proposal
...
2026-10-28 18:50:03 run finished: exit 0, 2s. 2 proposals in proposals/2026-10-28-1850.csv, 2 for a person to see
2026-10-28 18:52:00 run started: qwen2.5:3b, breaker 30%
...
2026-10-28 18:52:01 run finished: exit 0, 1s. 2 proposals in proposals/2026-10-28-1852.csv, 2 for a person to see
(The eight unchanged towns in each run are left out.) It worked, and it showed a problem none of the manual runs had.
The job that asked twice
Every run wrote the same two proposals again, into a new file. Run by hand once, that's fine. On a schedule every two minutes, it's thirty copies of the same question an hour, and the person reviewing them stops reading. A job that runs again and again has to remember what it has already asked.
So job.py now keeps one file, proposals/pending.csv, and doesn't propose anything that's already waiting there. Two runs, back to back:
2026-10-28 19:02:27 run finished: exit 0, 3s. 2 new proposals in proposals/pending.csv, 2 for a person to see
2026-10-28 19:02:27 run started: qwen2.5:3b, breaker 30%
2026-10-28 19:02:27 preflight ok: Ollama 0.40.11, qwen2.5:3b 357c53fb659c
2026-10-28 19:02:27 gate ok: every must-pass case is right
2026-10-28 19:02:27 Cedar Bend Dana Whitfield already proposed, still waiting for a person
2026-10-28 19:02:27 Elm Ridge Carla Mendes already proposed, still waiting for a person
...
2026-10-28 19:02:27 run finished: exit 0, 1s. 0 new proposals in proposals/pending.csv, 0 for a person to see
(The end of the first run and the eight unchanged towns are left out.) The second run checked everything, found the same answers, and asked nobody anything new. Running a job twice should leave things the same as running it once. That property has a name, idempotence, and it's what makes a job safe to run on a schedule, or to run again after a failure.
Remove the schedule when you're done, and check it's gone:
crontab -l | grep -v local-llm-practice | crontab -
crontab -l
A laptop isn't a server
Cron on a laptop has one big catch: when the lid is closed, the laptop sleeps, and nothing runs. Cron doesn't catch up on missed runs either. That's fine for practice, and it's the main reason scheduled jobs live on a machine that stays on.
Wherever a job runs, the most dangerous failure isn't one it reports. It's the run that never starts: the machine was asleep, the scheduler was off, the script lost permission to run. Nothing errors, so nothing is logged, and silence looks exactly like "nothing changed". python3 job.py status reports how long ago the last run finished, which is the start of the answer: something has to notice when that number gets too big.
In my real setup
CivicGrid's weekly mayor check runs from cron on my homelab, which stays on. Most of this lesson's job is a small copy of it. A lock stops two runs overlapping, which matters because a full re-read of every city takes hours. The circuit breaker halts a run when more than 2% of the cities checked have changed, once at least 25 have been checked. Proposals go to an admin page, and nothing changes a leader's name without me approving it. A run that fails posts a message to Slack. And it avoids asking twice in two ways: it skips pages that haven't changed since the last run, and it never proposes a name I've already rejected.
The silent failure has its own guard, a dead man's switch. Each run pings an outside service, Healthchecks.io, when it starts and when it ends. If the ping doesn't arrive on time, that service alerts me. It's there for exactly the failures nothing can log: the machine asleep, cron not running, or the script unable to run at all, which really did happen to me in October when it lost its permission to run.
Building this lesson showed me two gaps. The real check doesn't pin its model: if the model behind the name changed, it would carry on with a model I'd never tested. And the regression cases run when I change the check, not before every run. This job does both, in a few lines each.
Spot the problem
Three situations. Work out what's wrong before you read the explanation.
The job that never failed. A team's nightly job has shown no errors for three months. Then someone notices the data hasn't changed in all that time either. What happened? The job stopped running, and a job that doesn't run logs nothing. No errors isn't the same as success. A dead man's switch, an alert when a run doesn't report in, catches it.
The update nobody made. A scheduled extraction job uses a model by name. One week, its answers get noticeably worse, though nobody changed the code. What's a likely cause, and the fix? The model behind the name was updated. Pin the model's digest, and have the job refuse to run when it changes, until the new model passes the test suite.
The helpful retry. A job fails halfway, and the scheduler is set to retry it immediately. Reviewers start finding every proposal twice. What's missing? Idempotence. A retry runs the same work again, so a job has to recognize what it already did. Here, that meant not proposing anything already waiting for a person.
Where you are now
A job that runs by itself has to look after itself. Before it starts: a lock, so runs don't overlap, and a preflight that checks Ollama is up and the model is the exact one you tested. Then a gate of must-pass cases, and a circuit breaker for changes too large to be real. It proposes rather than writes, leaves a log and an exit code every time, and remembers what it already asked, so running it twice is the same as running it once. Every one of those protections was tested here by breaking the thing it protects. Lesson 13 steps back to the question the whole track has been circling: when is a small local model the right tool, and when is a large hosted one?
Check yourself
Why does the job pin the model's digest?
What is a circuit breaker for?
Why does each way of stopping have its own exit code?
Why did the lock exit with 0 when it skipped a run?
The scheduled job proposed the same two changes every run. What was missing?
Which failure can only a dead man's switch catch?
Key terms
- Service: a program that runs in the background, waiting for requests.
- Exit code: the number a program returns when it ends; 0 means success.
- Pinning: refusing to run with anything but the exact version you tested.
- Preflight: checks a job makes before it starts real work.
- Circuit breaker: a limit that stops a run when changes are too many to be real.
- Lock: a marker that stops two runs of the same job overlapping.
- Idempotence: running something twice leaves things as running it once would.
- Dead man's switch: an alert that fires when an expected signal doesn't arrive.
Go deeper
- The practice set (opens in a new tab):
job.pyandcurrent.csv. Add arejected.csvthat a person fills in, and make the job never propose a name that's in it, as the real check does. - Try the gate: pull
qwen2.5:7b, change the model injob.json, runpin, thenrun. Does it stop at the gate with exit 3? Remove 7B afterwards. - The Python track's Alerts and settings, for sending yourself an alert when a run fails.
- Healthchecks.io (opens in a new tab), one of several services that act as a dead man's switch for scheduled jobs.