Evaluate Your RAG Chatbot: Test Retrieval, Answers, and Citations

Your PDF RAG chatbot answers the first question correctly, quotes the handbook, and looks ready to share. Then a colleague asks the same thing in different words and gets the wrong page. One familiar question won't tell you which part failed or how often. To evaluate a RAG chatbot properly, you need a small set of questions with expected evidence, and a way to inspect retrieval separately from the answer.

We'll build an evaluation harness for the chatbot from the PDF RAG tutorial. It records which pages search returns, checks a few expected facts, validates quotes through the application, and keeps unsupported questions separate.

The harness lives in a separate download, the AI blog starters, which ships its own copy of the chatbot as pdf_rag.py. That file is the same code as rag.py in the PDF RAG starter, so you can run everything from one folder. To evaluate the project you already built instead, copy evaluate_rag.py into your pdf-rag-starter folder and change import pdf_rag as rag to import rag. The sample is one fictional four-page handbook, so treat its numbers as a check of this example, not an estimate for a real collection.

Decide what counts as a useful answer

Start by writing down the question, relevant source pages, expected facts, and whether the system should decline to answer. “What is the hotel limit?” needs the travel page and EUR 160, while a question about company revenue has no supporting passage in the sample. Labelling before a run stops you from moving the goalposts after seeing whatever the model produced.

The starter includes five cases: annual leave, hotel costs, equipment allowance, the lost-laptop deadline, and missing revenue information. These cases are intentionally easy to verify. For your own PDFs, add rephrased questions, exceptions, multi-section answers, and plausible questions the documents don't cover. If you use some questions to tune chunk settings or the prompt, keep a separate held-out set for the final comparison.

case = {
    "question": "What is the hotel reimbursement limit per night?",
    "pages": [2],
    "facts": ["160"],
    "abstain": False,
}

Measure RAG retrieval before generation

If the right passage doesn't reach the model, a bad final answer doesn't tell you much about the answer prompt. For each question, run retrieval and compare the returned source locations with your relevance labels. Our example labels physical page numbers because there is only one PDF and one passage per page. In a real collection, use a stable source identifier plus a location or chunk identifier; page 2 from two different files is not the same evidence, and several chunks can come from one page.

We'll record Hit@k, recall at k, and reciprocal rank for k equal to one, two, and four. Hit@k asks whether at least one relevant source appears in the first k results. Recall asks what fraction of the expected sources appeared. Reciprocal rank is one divided by the position of the first relevant source, or zero when none appears. They answer different questions: one useful passage first isn't enough for a request that needs two sections.

expected = {2}
returned = [3, 2, 1, 4]
k = 2
hits = expected.intersection(returned[:k])
hit_at_k = int(bool(hits))           # 1
recall_at_k = len(hits) / len(expected)  # 1.0
first_relevant_rank = next(
    (i + 1 for i, page in enumerate(returned[:k]) if page in expected), None
)                                        # 2
reciprocal_rank = 1 / first_relevant_rank if first_relevant_rank else 0  # 0.5

Try it below. The first ranking is the list from this snippet; the second is a recorded search for the hotel question. Change k or the relevance labels and watch which metrics move.

Don't give missing-answer cases a retrieval score

A question that the documents don't answer has an empty relevance set. Dividing by its size would be undefined, and calling retrieval successful just because it returned some text would be misleading. Our harness writes null values for those retrieval metrics and checks the application's final behaviour separately. Search will still return the nearest passages, because there's no calibrated no-answer detector at retrieval time.

The sample also makes Recall@4 unusually easy: there are only four passages, so returning four includes the entire collection. A perfect value there says very little; Hit@1 and the rank of the relevant page are the useful checks. Add realistic distracting passages before comparing retrieval methods or embedding models, and report the collection size beside any score.

Check answer facts without pretending to prove correctness

For supported questions, the harness checks for a narrow expected fact such as the number 160, using word boundaries so 160 doesn't accidentally match a longer token. That is useful as a regression signal, not a semantic correctness test. “The limit is not 160” contains the same number, and “one hour” can express the expected deadline without the digit 1. So keep the returned answer in the report and read it.

import re
 
def fact_check(answer, facts):
    return all(
        re.search(r"(?<!\w)" + re.escape(fact) + r"(?!\w)", answer)
        for fact in facts
    )

Separate the checks in your report so an evaluator can see what passed. The source page can be correct while the answer's conclusion is wrong; the expected number can appear while an exception disappears. The starter records expected behaviour and quote validation separately and sets human_claim_review to pending. That's deliberate: the string checks can't make that judgement, and an unreviewed row shouldn't quietly count as a pass.

Validate citations through the application

The PDF chatbot already rejects source IDs outside the retrieved set and quotations that don't occur in the named passage after whitespace normalization. The evaluator calls that same answer function rather than bypassing validation just to collect a score. If validation fails, the report records an error. If there is no answer or no citation, the application returns its no-answer message. Testing the real path catches the failures users would actually hit.

Quote existence still doesn't establish that the quote supports the answer. A response saying the hotel limit is EUR 999 can quote the real EUR 160 sentence, and the literal quote check will accept those words. The PDF RAG starter's tests/test_rag.py includes exactly that counterexample (test_real_quote_can_still_support_a_wrong_answer), and this harness's tests/test_evaluation.py adds the same lesson for fact matching. Research such as RAGAS examines separate aspects of retrieval and generated answers; our small harness keeps that separation but uses direct labels and narrow checks rather than claiming to implement RAGAS or a general-purpose factuality judge.

Run the harness on the sample handbook

Unzip the starter, install its main requirements in a virtual environment, start Ollama, and download nomic-embed-text and qwen3:4b (new to Ollama? see how to run LLMs locally). The harness builds a fresh index from its included PDF before running the cases, so it doesn't accidentally use an earlier saved version from another experiment. Each row is printed as JSON and written to rag-evaluation-results.json in the starter folder. Run from a writable copy of that folder, since both the fresh index and the result file are saved there.

unzip ai-blog-starters.zip
cd ai-blog-starters
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
ollama pull nomic-embed-text
ollama pull qwen3:4b
python evaluate_rag.py

In our recorded run, the four supported cases retrieved their labelled page first, returned the expected fact, and passed quote validation. The revenue question produced the expected no-answer response. The recorded JSON has every row, each still marked for human claim review. Five easy questions are a smoke test, not evidence of reliability on contracts, papers, or manuals.

Use failures to choose the next change

Suppose the hotel question returns the equipment page first. Print the extracted travel text and inspect its chunk before replacing the chat model. A missing header or split exception may explain the failure. If retrieval finds the correct passage but the model invents an amount, compare the prompt, returned answer, and quote. If an updated policy never appears, rebuild the index and check document versions. These are different failures. Change everything at once and you won't know which fix worked.

For a controlled comparison, save the same questions and expected sources, change one setting, and rerun. Keep extraction settings, embedding model identifiers, k, generation options, and model/server versions with the output. Wall-clock time can include loading or queueing, so repeat timing measurements if speed matters. Then confirm the winner on the held-out questions, not only the ones you debugged with. When retrieval looks fine and the model still struggles with your domain's wording, that's the point where RAG vs fine-tuning becomes a real question.

Treat automatic judges as another component to evaluate

A model-based judge can help review more answers, but its output introduces another set of possible mistakes. It may reward polished wording, miss a contradiction, or react differently when answer text contains instructions. If you add one, give it the question, answer, and relevant evidence, define the judgement precisely, and compare it with human reviews on a sample. A single overall score hides whether it judged retrieval, evidence support, completeness, or just plausibility.

You don't need an elaborate evaluation platform to begin. A handful of carefully labelled questions, the returned passages, the generated answer, and a report you can read are enough to make debugging more disciplined. Expand that set as new failures appear, while preserving some questions for a less biased comparison. The goal is to know what a change improved and what remains unchecked. The same habit applies when your chatbot starts calling tools: see building an AI agent with tool calling.