Ask your own documents: embeddings and retrieval

Where you are

Until now, you've always handed the model the page. One town, one page, one question. Real questions don't come with the page attached. "Can I burn my garbage?" is somewhere in a pile of documents, and you don't know which one.

You could paste the whole pile into the prompt. Lesson 6 showed why that stops working: the context window is a fixed size, and anything that doesn't fit is silently cut. A pile of a few thousand documents will never fit.

The fix is to find first, then answer. Search the pile for the few passages most likely to hold the answer, and give the model only those. That's retrieval-augmented generation, or RAG: the model's answer (the generation) is helped (augmented) by a search step (retrieval) that runs first.

The practice pile

The practice set has a new folder, docs/, with twelve short Larkspur County documents: trash rules for two towns, parking rules for two others, council minutes, water rates, the library, the pool, and more. About 1,300 words in all, so small enough to check by hand.

questions.csv has 18 questions with known answers, and several are traps:

Three questions have no answer in the documents at all. The right reply to those is NOT IN DOCUMENTS.

Embeddings: meaning as numbers

The old way to search is to match words: find the passage that shares the most words with the question. It works until the question uses different words from the page. "Garbage" doesn't match "trash", and "TV" doesn't match "televisions".

An embedding solves that. An embedding model reads a piece of text and turns it into a long list of numbers that represents its meaning. Texts that mean similar things get similar lists, even when they share no words. To compare two texts, you compare their lists, with a calculation called cosine similarity: 1 means the same direction, 0 means unrelated.

Embedding models are a different kind of model from the one that writes answers, and much smaller. This lesson uses nomic-embed-text:

ollama pull nomic-embed-text

pulling 970aa74c0a90: 100% ▕███████████████████████████████████████████████████████████████████████████████████████████▏ 274 MB
pulling 31df23ea7daa: 100% ▕███████████████████████████████████████████████████████████████████████████████████████████▏  420 B
verifying sha256 digest
writing manifest
success

At 274MB, it's a fraction of the size of qwen2.5:3b. Now split the documents into passages and embed every one:

python3 rag.py index

39 passages from 12 documents, 768 numbers per passage, saved to index.json

Each passage is one paragraph, and each is now 768 numbers. Ask the question with no town and no matching words:

python3 rag.py search "Can I burn my garbage?"

0.695  willow-bend-burning.md               Only leaves, brush and untreated wood may be burned. Burning trash, plastic or building ma...
0.639  willow-bend-burning.md               Burning leaves and brush in your yard needs a burning permit from the Willow Bend Fire Dep...
0.636  elm-ridge-trash.md                   Electronics, such as televisions and computers, are not accepted in trash or recycling. Th...

The closest passage is exactly the right one, "Burning trash… is against the law", and the question never said "trash".

Testing retrieval on its own

Retrieval is a step of its own, so it gets a test of its own: for each answerable question, does the right document come back first? retrieve compares word search with meaning search:

python3 rag.py retrieve

== words: right document first 13 of 15, in the top 3 13 of 15
   Where can I recycle an old TV?
      wanted elm-ridge-trash.md, top was county-dog-license.md
   Can I burn my garbage?
      wanted willow-bend-burning.md, top was alder-creek-minutes-2026-09.md

== meaning, with titles: right document first 15 of 15, in the top 3 15 of 15


== meaning, without titles: right document first 14 of 15, in the top 3 15 of 15
   Who ran the Alder Creek council meeting in September?
      wanted alder-creek-minutes-2026-09.md, top was birch-hollow-select-board-2026-10.md

Three findings:

• Word search did well, until the words changed. It found 13 of 15, mostly because the questions name the town and the documents do too. Its two misses were the two questions that name no town and use different words from the page. Old-fashioned search is a real option for a small, tidy pile.
• Meaning search found all 15. Including "garbage" and "TV".
• Titles matter. When passages are split into paragraphs, a paragraph like "Present: Council President Diego Ramirez…" no longer says which town it's from. So rag.py puts the document's title at the top of every passage. Without titles, the Alder Creek minutes lost to Birch Hollow's: both are meeting minutes that start with who was present.

Four ways to answer

Now the whole job. answer tries four ways, and scores every reply:

• none: no documents at all. Does the model know, or guess?
• all: every document pasted into the prompt. The pile is small enough to fit.
• top3: only the three passages closest to the question.
• top1: only the single closest passage.

The prompt tells the model to answer only from the documents, and to reply NOT IN DOCUMENTS if they don't contain the answer.

python3 rag.py answer

The output lists every question, the reply and its score, for each mode. The totals:

(The top3 row is after the scoring fix described below; the first run scored one more reply as made up.)

With no documents, the model didn't guess. It said NOT IN DOCUMENTS to all 18, which happens to be right for the three that have no answer. That's the prompt's escape hatch working, and it's helped by the county being made up: there was nothing to half-remember. Compare Lesson 2, where a model with no page to read invented a mayor for Maple Falls.

Every document in the prompt got 17 right, and one dangerous wrong:

   WRONG          How much is a family season pass for the Stone Mill pool?
                  A family season pass for the Stone Mill pool costs $60.

That $60 is Quarry Hill's pool. With every document in front of it, the model found a pool, found a family pass, and attached it to the wrong town.

Three passages did worse than one. top3 missed two questions that top1 answered:

   missed         Where can I recycle an old TV?
                  NOT IN DOCUMENTS
   missed         Can I burn my garbage?
                  NOT IN DOCUMENTS

Retrieval had put the right document first for both. The model had the answer in front of it, with two less relevant passages beside it, and said it wasn't there. More context isn't automatically better: the extra passages seem to have made a small model less sure, though I can't see inside it to know why.

The closest passage alone got 17 right and nothing dangerous. Its one miss is worth a look:

   missed         Can I burn my garbage?
                  NOT IN DOCUMENTS

The search above put "Burning trash… is against the law" first for this question. So the model was handed exactly the right passage, and still said it wasn't there. Finding the right passage and answering from it are two different steps, and either can fail. That's why retrieve and answer are tested separately: one tells you whether the search worked, the other whether the model used what it found.

Try every question with every mode:

The trap: it always finds something

Look at the Stone Mill pool question in top3:

   WRONG          How much is a family season pass for the Stone Mill pool?
                  A family season pass costs $60 for residents at the Quarry Hill Community Pool.

There is no Stone Mill pool document. But a search never comes back empty: it returns the closest passages, whatever they are. The closest passages to "Stone Mill pool" were about Quarry Hill's pool, so that's what the model got. It even said "Quarry Hill" in its answer, and the answer is still wrong for the question that was asked.

This is the most important thing to know about RAG. Retrieval answers "what's most similar?", not "is the answer here?" For a question the documents can't answer, the most similar passage is usually about something nearby: another town, another year, another pool.

Lesson 8's idea helps here: check the answer against something you know, in plain code. The question names Stone Mill; the passage came from a Quarry Hill document. A few lines that compare the town in the question with the town in the passage's title would have caught it. Every system that retrieves needs a check like that, because the search itself never says "nothing found".

My own scoring mistake

The first run scored one reply as MADE UP:

   MADE UP        When is leaf collection in Cedar Bend?
                  Yard waste is collected with the trash from April through November in Cedar Bend.

I'd written that question as having no answer, because no document mentions leaf collection. But Cedar Bend's trash page does say yard waste goes out with the trash from April to November, and leaves are yard waste. A person would accept that answer. The test was unfair, not the model.

So I changed the case to accept either NOT IN DOCUMENTS or an answer about yard waste. Lesson 10's rule paid off immediately: rag.py saves every reply to results/, so rescore re-scored the saved answers with the fixed case, without running the model again:

python3 rag.py rescore top3

   right          When is leaf collection in Cedar Bend?
                  Yard waste is collected with the trash from April through November in Cedar Bend.
   right 15, missed 2, WRONG 1

(The other sixteen questions are left out; their scores didn't change.)

Writing good test cases is part of the work. A case that's ambiguous to a person will be ambiguous to a model too.

When to use RAG, and when not to

Put together, the results give a fairly plain guide:

• If the documents fit in the context window, try pasting them all first. Here, that got 17 of 18. It's simpler, and there's no search to go wrong.
• When they don't fit, retrieve. Meaning search found the right document every time here. Word search is a reasonable start for a small, tidy pile.
• Put the title on every passage, so a paragraph still says where it came from.
• Test retrieval and answering separately. Either one can fail while the other works.
• Fewer, closer passages can beat more. On this model, one passage beat three. Measure it on your own documents.
• Check the source against the question. Retrieval always returns something, so a wrong-town answer looks just as confident as a right one.

Spot the problem

Three situations. Work out what's wrong before you read the explanation.

The search that always succeeds. A town's chatbot uses RAG over its documents. Someone asks about a recycling fee the town doesn't charge, and the chatbot confidently quotes the neighboring county's fee. Nothing in the logs shows an error. What went wrong? Retrieval always returns the closest passages, and the closest ones were about another place. The fix isn't a better search; it's a check that the passage actually applies to the question, and a way to say "not found".

Testing only the final answer. A team scores their RAG system on answers alone, and it gets 80% right. They try a better embedding model, and the score doesn't move. What might they be missing? Whether the right passage was retrieved at all. If retrieval already finds the right passage, a better embedding model can't help, and the problem is the answering step. In this lesson, "Can I burn my garbage?" was retrieved perfectly and still missed.

Paragraphs without a home. A team splits a hundred council minutes into paragraphs for retrieval. Questions about one city keep getting answers from another. What's the likely cause? A paragraph like "The motion passed 4 to 1" doesn't say which city it's from. Adding the document's title to every passage gives the search, and the model, the missing context.

Where you are now

RAG finds first, then answers. An embedding model turns each passage into numbers that capture its meaning, so a search can match "garbage" to "trash". Meaning search found the right document for all 15 answerable questions, word search for 13, and leaving titles off passages cost one. Answering from the closest passage alone got 17 of 18 with nothing dangerous; three passages did worse, and pasting every document in got one dangerous wrong town. Retrieval and answering fail separately, so test them separately, and never forget that a search always finds something. Lesson 12 takes everything built so far and turns it into a service: one that runs on a schedule, by itself, on a machine you don't sit in front of.

When you're done with this lesson, remove the embedding model:

ollama rm nomic-embed-text

Check yourself

Key terms

• RAG (retrieval-augmented generation): find the passages most likely to hold the answer, then answer from those only.
• Retrieval: the search step, which picks the passages to give the model.
• Embedding: a list of numbers representing a text's meaning; similar meanings get similar lists.
• Embedding model: a model that turns text into embeddings, rather than writing answers.
• Cosine similarity: a way to compare two embeddings; closer to 1 means more similar.
• Passage (or chunk): a piece of a document, here one paragraph, retrieved on its own.
• Index: the saved embeddings for every passage, so the documents don't have to be embedded again for each question.

Go deeper

• The practice set (opens in a new tab): rag.py, docs/ and questions.csv. Add the code check from "The trap" above: reject an answer when the question names a town and the passage's title names a different one. Then rescore.
• Try top2: change top3 to top2 in the command, since the mode name sets how many passages are used. Does it land between one and three?
• Retrieval-augmented generation (opens in a new tab) on Wikipedia, for the wider picture.
• Lesson 6's Context windows, for why the whole pile can't simply go in the prompt.