How to build a RAG system over your own documents that cites the page and stops making things up

RAG stands for retrieval augmented generation. Instead of asking the model to remember our documents, we search them first, paste the best passages into the prompt next to the question, and ask the model to answer from those passages and cite them. It is the right tool when the documents are too big for the context window, change often, or cost too much to send with every question.

It is not always the right tool. As of 28 October 2026 Claude Sonnet 5.5, Haiku 5.5 and Opus 5.5 all take 1M tokens of context. At roughly 500 tokens a page that is about 2,000 pages. A 40 page handbook or a handful of contracts fit with room to spare. For those we skip the search layer, send the documents as document blocks with citations turned on, cache them, and get page-level citations with no chunking to get wrong. Step 3 puts the costs side by side so we can choose with numbers.

The build below uses two vendors on purpose. An OpenAI vector store does the search, because it chunks and embeds for us and we do not have to run a database. Claude writes the answer, because its citations API returns the exact quoted text from the passages we supplied, so a citation cannot point at something we never sent. If we want one vendor, step 5 is the pure OpenAI path and it works on its own.

We need Python 3.11 or newer, an OpenAI API key and an Anthropic API key. Both consoles bill by usage, so no subscription is needed. Put the keys in the shell, never in the code, and never in a notebook we might share.

Prices as of 28 October 2026, per million tokens in and out: claude-sonnet-5-5 $2 and $10, claude-haiku-5-5 from $0.10 and $0.50, claude-opus-5-5 $4 and $20. On the OpenAI side gpt-6.1-terra is $1 and $5 and gpt-6-luna is $0.10 and $0.50. Embeddings with text-embedding-3-small cost $0.02 per million tokens. Vector store storage is free for the first GB and $0.10 per GB per day after that, and each file search call costs $2.50 per thousand calls. Check the pricing pages before we quote these to anyone, they move.

Prepare the documents while pip runs. PDFs must contain real text, not scans, or the chunker gets nothing. One topic per file where we can split them. Give files names that mean something, because the filename is what comes back in the citation.

Two numbers decide most of the cost and most of the quality: chunk size and overlap. The OpenAI vector store defaults to 800 token chunks with a 400 token overlap. max_chunk_size_tokens can be anything from 100 to 4096, and the overlap can be at most half the chunk size or the request is rejected. Small chunks give precise citations but split a table or a clause across two pieces, and the answer falls into the gap. Large chunks keep clauses whole but every query carries more tokens.

Tokens per query is the number to watch. Each question sends the top results, ten by default, times the chunk size, plus the system prompt and the question. At 800 token chunks that is about 8,400 input tokens a question. On Sonnet 5.5 that is under two cents, before the answer.

The calculator uses Sonnet 5.5 at $2 in and $10 out, ten results a query, a 500 token answer, and cache reads at 10 percent of the input price. Anthropic lists cache reads at 5 percent for Sonnet 5.5 and Opus 5.5 as of 28 October 2026, so the cached figure may be lower in practice. We kept 10 percent because that is the rate on the other models and it is the safer number to plan with.

Three calls: create the store, upload each file with purpose="assistants", then attach the file to the store with the chunking we chose. Attaching is asynchronous, so we poll until the status is completed. A status of failed almost always means the PDF had no extractable text.

Limits as of 28 October 2026: 512 MB and 5 million tokens per file. A 5,000 page scan will not go in as one file, and should not anyway, because a citation to a 5,000 page file is not a citation anyone can check.

import time
from openai import OpenAI
client = OpenAI()

store = client.vector_stores.create(name="handbook-2026")
for path in ["docs/handbook-2026.pdf", "docs/leave-policy-2026.pdf"]:
    with open(path, "rb") as f:
        up = client.files.create(file=f, purpose="assistants")
    vf = client.vector_stores.files.create(
        vector_store_id=store.id, file_id=up.id,
        chunking_strategy={"type": "static", "static": {
            "max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}})
    while vf.status in ("in_progress", "queued"):
        time.sleep(2)
        vf = client.vector_stores.files.retrieve(up.id, vector_store_id=store.id)
    print(path, vf.status)
print("store:", store.id)

Save the store id somewhere we will find it again. We re-use it for every question and we do not want to pay to ingest twice.

This is the whole OpenAI path. We call the Responses API with a file_search tool that points at our store. The answer comes back as a message, and each cited passage is an annotation of type file_citation carrying the file_id and filename. Add include=["file_search_call.results"] to get the retrieved chunks back as well, which is what we read when an answer looks wrong.

r = client.responses.create(
    model="gpt-6.1-terra",
    input="What is the notice period when an employee resigns?",
    tools=[{"type": "file_search", "vector_store_ids": [store.id],
            "max_num_results": 8,
            "ranking_options": {"score_threshold": 0.5}}],
    include=["file_search_call.results"])
for item in r.output:
    if item.type == "message":
        for part in item.content:
            print(part.text)
            for a in part.annotations:
                if a.type == "file_citation":
                    print("cites:", a.filename, a.file_id)

max_num_results defaults to 10 and goes up to 50. ranking_options.score_threshold runs from 0 to 1 and drops weak matches. Set it, because an empty result set is what lets the model say "not in the documents" instead of answering from a chunk about something else. Note what this path does not give us: a page number. The citation names the file. If we need pages from OpenAI, we split PDFs per page before upload or keep our own page map. Step 6 is the way we get real page numbers.

Claude's citations API is the reason we use two vendors. We pass each retrieved chunk as a search_result block with a source, a title and the text, and set citations: {enabled: true}. Claude's answer then comes back as text blocks, each carrying a list of citations with the cited_text and the index of the block it came from. The quoted text is pulled from what we sent, and it is free of output tokens.

import anthropic
claude = anthropic.Anthropic()

blocks = [{
    "type": "search_result",
    "source": f"{c['file']}#page={c['page']}",
    "title": f"{c['file']}, page {c['page']}",
    "content": [{"type": "text", "text": c["text"]}],
    "citations": {"enabled": True},
} for c in chunks]

msg = claude.messages.create(
    model="claude-sonnet-5-5", max_tokens=800, system=SYSTEM,
    messages=[{"role": "user", "content": blocks + [{"type": "text", "text": question}]}])
for b in msg.content:
    if b.type == "text":
        print(b.text)
        for c in (b.citations or []):
            print("  cites:", c.title, "|", c.cited_text[:80])

The chunks list is whatever our retriever returned, with the page we recorded at ingest. If we skipped RAG in step 1, we send the PDF itself as a document block with citations: {enabled: true}, and the citations come back as 1-indexed page ranges straight from the PDF. Plain text documents cite sentence ranges and custom content blocks cite the block indices we gave. The system prompt below is the one we use. It is short because the citations do the enforcing, not the adjectives.

Every question in step 6 resends the same system prompt, and in the no-RAG version the same documents. Prompt caching charges those tokens once at a small premium and then reads them at 10 percent of the input price on later calls (5 percent on Sonnet 5.5 and Opus 5.5 as of 28 October 2026). The cached part must be at least 512 tokens, and it must sit at the start of the prompt and be byte for byte the same each time.

Mark the last document block, or the last stable block, with cache_control:

docs[-1]["cache_control"] = {"type": "ephemeral"}
msg = claude.messages.create(model="claude-sonnet-5-5", max_tokens=800,
    system=SYSTEM, messages=[{"role": "user", "content": docs + [q]}])
print(msg.usage.cache_creation_input_tokens, msg.usage.cache_read_input_tokens)

The first call shows tokens under cache_creation_input_tokens, the second under cache_read_input_tokens. If reads stay at zero, something above the marker changed between calls. A timestamp in the system prompt is the usual culprit.

With RAG the retrieved chunks differ per question, so only the system prompt caches, and that is small. Caching pays most in the no-RAG design, where the whole corpus is the stable prefix. That is why a 40 page handbook sent whole and cached is cheaper than a vector store, and a 2,000 page archive is not.

A citation is only worth something if a reader can open the page and find the sentence. Two things break that, and a prompt fixes neither.

The first is wrong page metadata from the loader. In October 2026 Open WebUI shipped a bug where the Docling engine stored the markdown piece index as the page number, so every citation pointed at the wrong page while looking perfect. Our own loader can do the same. The check: for each chunk, open the PDF at the recorded page and confirm the first line of the chunk is on it. Do this on ten random chunks after every ingest.

The second is a citation to a chunk that was never in the context. The prompt says "only use the context" and the model writes a plausible page number anyway. Researchers call it citation hallucination and it survives system prompts. With Claude's citations API the citation carries a block index, so the check is code: the index must exist in the list we sent and cited_text must be a substring of that block. With the OpenAI path, the file_id must be one of ours. Reject the answer when either check fails and show the reader "Not in the documents" instead. A refusal is a smaller failure than a confident wrong page.

Before anyone else uses it, we write 30 to 50 questions with the correct answer and the file and page that holds it. Real questions from real people, including five that the documents do not answer at all. Then we run them all and score three things on their own, because they fail for different reasons and get fixed in different places.

Retrieval hit: did a chunk from the right page land in the results? If not, the model never had a chance. Fix with chunk size, overlap, more results or better file names. Groundedness: does every claim in the answer appear in the retrieved text? A cheap judge model, Haiku 5.5 with the chunk and the answer, scores this well enough. Fix with the system prompt and the score threshold. Citation correctness: does the cited page contain the quoted text? This is the step 8 check run across the set. Fix in the loader and the page map.

Run it after every change to chunking, prompts or models. A change that lifts groundedness and drops retrieval hit is not an improvement, and only three separate numbers show that. Keep the eval file in the repo next to the code.