Local or hosted?

Where you are

This whole track has used small models on your own machine. That was a choice, and Lesson 1 said the trade plainly: local models are smaller and less capable than the biggest hosted ones, but free to run, private, and they work without the internet.

Twelve lessons later, you have the tools to test that trade instead of taking it on trust: a job that records exactly what it does, a test suite that compares two setups case by case, and real numbers from real runs. This lesson puts them to work on the last question: when should a job like this run on a hosted model instead?

How hosted models are priced

A hosted model runs on someone else's computers. You send your prompt over the internet through an API (application programming interface), and pay for what you use. The price is per token, the pieces of text a model reads and writes, and it's quoted per million tokens, with two prices: one for the tokens you send (input) and a higher one for the tokens the model writes back (output).

Providers usually offer a range of models, from small, fast and cheap to large, slow and expensive. Anthropic's current Claude models, from its pricing page in October 2026:

(Haiku 5.5's prices are for prompts up to 100,000 tokens; longer prompts cost more.) There's also a batch option: send many requests at once, accept results within a day instead of seconds, and pay half price. A weekly job that nobody waits on is exactly what batch is for.

Prices change, and other providers price their own models differently, so treat this table as a snapshot. The method below works with any price list.

Measure your own job

Guessing what a job costs is how bills surprise people. So costs.py in the practice set measures it: it runs Lesson 10's 16 cases on your local model and records, for each one, how many tokens went in, how many came out, and how long it took. Ollama reports all three with every answer.

python3 costs.py

case                   tokens in tokens out  seconds
Cedar Bend                   226          5     1.53
Elm Ridge                    214          5     0.12
Alder Creek                  268          5     0.15
Birch Hollow                 221          2     0.10
Fox Run                      344          4     0.18
Juniper Flats                213          3     0.10
Heron Point                  238          6     0.14
Quarry Hill                  256          2     0.11
Willow Bend                  218          6     0.12
Stone Mill                   210          3     0.10
Larch Grove                 6622          4     3.69
Cedar Bend (planted)         259          2     0.12
Elm Ridge (planted)          235          4     0.13
Alder Creek (planted)        268          5     0.04
Fox Run (planted)            364          2     0.07
Stone Mill (planted)         235          3     0.08
average                      649        3.8     0.42

Three things stand out.

• Almost all the tokens go in. A page is a few hundred tokens; the answer is a name, about four. For a job that reads a lot and writes a little, the input price decides the bill.
• One long page moves the average a lot. Most pages are 200 to 360 tokens. Larch Grove, Lesson 6's long page, is 6,622, and it alone pulls the average up to 649. Real pages vary like this, so measure on pages like your real ones.
• The first answer is the slow one. Cedar Bend took 1.53 seconds because the model was being loaded into memory, as Lesson 12 described. Most answers after that took about a tenth of a second.

One caution: these are Ollama's token counts. Every model splits text into tokens its own way, so a hosted model's count for the same page will differ somewhat. The costs below are estimates, not quotes.

What it would cost

costs.py then scales the average up to a CivicGrid-sized job: all 3,064 cities, re-read four times a month, which is the worst case, since a real weekly run skips pages that haven't changed:

Estimate: 3,064 pages a run, 4 runs a month = 12,256 pages
   8.0 million tokens in, 0.05 million out
   local, qwen2.5:3b: about 1.4 hours of this machine's time

hosted model          per month  batch, half price   prices checked
Claude Haiku 5.5          $0.82              $0.41   2026-10-28
Claude Sonnet 5.5        $16.39              $8.19   2026-10-28
Claude Opus 5.5          $32.77             $16.39   2026-10-28
Claude Fable 5.1         $81.93             $40.97   2026-10-28

And a more typical month, about 200 changed pages a run:

python3 costs.py --pages 200 --runs 4

Estimate: 200 pages a run, 4 runs a month = 800 pages
   0.5 million tokens in, 0.00 million out
   local, qwen2.5:3b: about 0.0 hours of this machine's time

hosted model          per month  batch, half price   prices checked
Claude Haiku 5.5          $0.05              $0.03   2026-10-28
Claude Sonnet 5.5         $1.07              $0.53   2026-10-28
Claude Opus 5.5           $2.14              $1.07   2026-10-28
Claude Fable 5.1          $5.35              $2.67   2026-10-28

(The table of cases printed again before each estimate; it's left out here.)

Practice pages are small, though. A real city page sends the model up to 12,000 characters, which at the common rule of thumb of about four characters per token is roughly 3,000 tokens, more than four times the practice average. Real checks also ask a second question for a second look. Worked through for the worst case, every city re-read four times a month with two questions each, that's about 74 million input tokens: roughly $7 a month on Haiku, $148 on Sonnet, $296 on Opus and $740 on Fable, half that with batch. Rough numbers, but the right size.

Move the settings and see:

So the honest finding is this: for a job like this, cost is the weakest argument for running locally. The cheapest hosted model costs pennies to a few dollars a month. The most capable ones cost real money at full scale, but nothing like a server. Local isn't free either: a machine that stays on, its electricity, and your time keeping it running. If cost were the only question, a small hosted model would win for most jobs this size.

Cost isn't the only question.

What money doesn't measure

Quality, which you have to measure. Hosted models, especially the large ones, are generally far more capable than a 3B model on your laptop. But "generally" isn't a measurement. Lesson 10 found that the bigger local model did worse on this job, falling for the mayor-elect trap that the smaller one passed. A hosted model gets the same treatment: add it to suite.py, run the sixteen cases, compare it case by case with your baseline, and look hard at every WORSE. Then you know.

Where your text goes. With a local model, the page and the question never leave your machine. With a hosted one, they travel to the provider's computers, under the provider's rules for how long they're kept and who can see them. For public city web pages, that hardly matters. For medical records, legal files, customer data or anything you've promised to keep private, it can decide the question on its own, before cost or quality come up.

Pinning, and models that retire. Lesson 12's job refuses to run unless the model is the exact file you tested, and with a local model you can keep that file for as long as you like. A hosted model changes on the provider's schedule: new versions arrive, and old ones are retired. Anthropic, for example, gives at least 60 days' notice before retiring a publicly released model. That's fair warning, but it means a hosted job will eventually need re-testing on a new model whether you planned a change or not. Pin the exact model version in your requests, keep the test suite, and expect to use it.

Working offline, and rate limits. A local model works with no internet connection and answers as fast as your hardware allows. A hosted model needs the network, and providers limit how many requests you can send in a given time. Neither matters much for a weekly job. Both matter for something that has to keep working when the connection doesn't.

Learning. Running a model yourself teaches you things a hosted API hides: context windows that silently cut your prompt, models that unload from memory, digests that change under a name. Every lesson in this track came from seeing one of those up close.

A way to decide

Put together, the questions to ask about any job, in order:

1. Can the text leave your machine? If not, it's local, and the rest is about making local good enough.
2. Has the job been tested on both? Run the same suite on a local and a hosted model, and compare case by case. Don't decide quality from model size or reputation.
3. What does it cost at your real volume? Measure your tokens on real pages, multiply by real volume, and include batch prices if nobody waits for the answer.
4. What happens when the model changes? Local: you choose when. Hosted: the provider eventually does. Either way, the test suite is your protection.
5. Does it have to work offline, or very fast, or at very high volume? Each pushes one way or the other.

Often the answer isn't one or the other. A common pattern is to run a cheap model on everything and send only the hard or uncertain cases to a bigger one: the same idea as Lesson 8's checks, where a cheap check runs first and the expensive one only runs when it's needed.

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

The estimate from the brochure. A team estimates a hosted model's monthly cost by multiplying the number of requests by a typical prompt size from a blog post. The first bill is four times the estimate. What did they skip? Measuring their own prompts. Here, practice pages averaged 649 tokens and real city pages are estimated at around 3,000. One long page can change the average a lot. Measure on real inputs.

The model that quietly moved. A team built an extraction job on a hosted model two years ago, tested it once, and never looked again. The provider retired that version and the job now uses its successor. Nobody noticed. What's the risk? The job is running on a model that was never tested on this job. Lesson 10 showed that a model change can turn right answers into dangerous wrong ones. Keep the test suite, and run it whenever the model changes, by your choice or the provider's.

Private data, cheap model. A clinic wants to summarize patient notes and picks a hosted model because it's cheaper and better than anything they can run locally. What question should come first? Whether the notes are allowed to leave their systems at all, under their own rules and the law. Privacy comes before cost or quality. If the answer is no, the choice is local, however the other numbers look.

Where you are now

That's the whole track. You started with what a language model is: a next-word predictor that sounds sure whether or not it's right. Since then, you've:

• run one on your own laptop, and picked a model your machine can fit;
• talked to it from code, and written prompts that do one narrow job;
• learned what silently happens when a page doesn't fit the context window;
• asked for structured answers, and checked them in code;
• caught made-up names, and defended against pages that give the model orders;
• built a test suite that refuses changes that make anything worse;
• searched a pile of documents by meaning;
• run the whole thing as a job that looks after itself.

And in this lesson, you measured what it would cost to run all of it hosted, and found that the deciding questions are rarely about money.

The same habit runs through every lesson: measure, don't assume. The 3B model's second look looked perfect until the right answers were counted. The bigger model did worse. The hardened prompt fixed two attacks and created a third. The test suite found a bug in itself. Every one of those came from running the thing and reading what actually happened, which is the most useful thing a person working with these models can do.

Check yourself

Key terms

• Hosted model: a model that runs on a provider's computers, used over the internet and paid for by use.
• API (application programming interface): the way a program sends requests to a service and gets answers back.
• Input and output tokens: the tokens you send, and the tokens the model writes back; priced separately.
• Batch: sending many requests at once for slower answers at a lower price.
• Rate limit: a cap on how many requests a provider accepts in a period of time.
• Retirement: a provider switching off an old model version.

Go deeper

• The practice set (opens in a new tab): costs.py and prices.csv. Update the prices from your provider's pricing page, and measure a few real pages of your own.
• If you have a hosted API key, add a hosted model to suite.py and compare it with your base run, case by case. Check the provider's pricing first: the sixteen cases cost very little, but it's your bill.
• Anthropic's pricing (opens in a new tab) and model deprecations (opens in a new tab) pages, as examples of what to read before building on a hosted model.
• Back to Lesson 1, What an LLM is, and isn't. It reads differently now.