RAG vs Fine-Tuning: Which One Does Your Project Need?

RAG vs fine-tuning is usually framed as a contest, but the two fix different problems. Say your company changes its hotel allowance from EUR 160 to EUR 180. You want the chatbot to answer with the new limit, cite the updated page, and stop quoting the old one. Now imagine a different problem: the facts are correct, but the chatbot keeps ignoring your required response format. Both look like "the model needs improvement," yet one is about access to current information and the other is about behaviour across many requests.

A project can need both. We'll use the same fictional handbook as the PDF chatbot tutorial, then compare what each approach would change. The EUR 180 revision is a hypothetical edit for this explanation; the downloadable handbook still says EUR 160, which matters when you run the code or check its citations. We'll finish with a small decision process you can apply to your own application's failures.

What RAG changes

RAG retrieves information before generating an answer. The application searches its documents, puts selected passages into the prompt, and asks the model to answer from them. Updating a policy therefore means updating the source document and rebuilding the affected index, not retraining the answer model. You still need to check that retrieval finds the new passage and that the answer uses it. A stale index can give a fluent, carefully cited answer to yesterday's policy question.

The original RAG paper studies combining retrieval with generation. Our small PDF example is a teaching application of that separation, not a reproduction of the paper's architecture or benchmarks. Its useful property is traceability: you can inspect the text sent to the model and show readers where the answer came from.

import json
 
messages = [
    {"role": "system", "content": "Use the provided passages only. Give a short answer followed by its source. If they do not answer the question, say so."},
    {"role": "user", "content": json.dumps({
        "question": "What is the hotel limit?",
        "passages": [{
            "source": "handbook.pdf", "page": 2,
            "text": "Hotel limit: EUR 160 per night; higher limits require written manager approval before booking."
        }],
    })},
]

What fine-tuning changes

Fine-tuning updates learned parameters using examples of the behaviour you want. With LoRA, those updates live in adapters while the original weights stay frozen. For a support router, you might train examples mapping messages to a fixed set of labels. For a specialist writing assistant, you might train a particular response style or task format. The LoRA paper describes an efficient way to represent updates, but it doesn't decide which training data is appropriate.

Training can influence factual knowledge as well as behaviour, so it would be wrong to say it can never teach a model facts. The practical problem is maintaining and verifying those facts when they change. A fine-tuned answer doesn't expose which training example supported it, and replacing a particular fact isn't equivalent to editing one document in a searchable collection. Try it below: change the policy and see what each system needs before it catches up.

RAG vs fine-tuning: start with the failure you can observe

Before choosing either approach, write down what went wrong in an actual request. If the model never saw the current policy, giving it the right passage may be the smallest useful change. If it saw the correct passage but answered with the wrong number, inspect its use of evidence rather than assuming more documents will help. If it answered correctly but broke a fixed JSON contract, try a schema and validation before starting a training job.

A prompt baseline is worth doing even when you expect to train later. Give the model a clear instruction, a few representative examples where appropriate, and the information it needs for the task. Measure the result on questions you didn't use to write the prompt; the RAG evaluation tutorial shows one way to build that question set. If the baseline is already sufficient, training would add data preparation and deployment work without addressing an observed gap.

Try a policy update without training

The starter's decision example keeps a fictional policy record separate from the instruction about response style. A small function builds the passage from the record, so editing the policy changes the evidence. This isn't a retrieval system by itself; it shows where a retrieved passage would enter the request. The complete PDF project adds extraction, embeddings, ranking, and checked quotes. When you edit its PDF, rebuild the saved index before expecting search to use the revised text.

def policy_passage(policy):
    return {
        "source": "handbook.pdf", "page": 2,
        "text": f"Hotel limit: EUR {policy['hotel_limit_eur']} per night; "
                f"higher limits require {policy['exception']}.",
    }
 
policy = {"hotel_limit_eur": 160, "exception": "written manager approval before booking"}
before = policy_passage(policy)
# A hypothetical revision changes the evidence, not the model weights.
policy["hotel_limit_eur"] = 180
after = policy_passage(policy)
print(before["text"])  # ... EUR 160 per night ...
print(after["text"])   # ... EUR 180 per night ...

Check the update with two kinds of questions: one asking for the amount directly and another asking about an exception or a previous version. If both old and new handbooks remain indexed, preserve effective dates and decide which versions each question may use. That is a document-management decision. Neither an embedding model nor an answer prompt can reliably infer your organization's version policy from two conflicting numbers.

Try stable behaviour with an adapter

The LoRA tutorial trains a small model to label support requests. Its data teaches an input-to-output pattern rather than a changing hotel allowance. In the recorded run, the model learned to emit labels but still misrouted one held-out shipping request. Formatting and correctness are different outcomes.

You can combine that kind of adapted behaviour with retrieval. An adapted model might follow a specialist response convention while the prompt provides current documents for a particular question. Test the combination, because training on one prompt style may not transfer cleanly to a request containing retrieved passages or unfamiliar document layouts. Keep retrieval results and final answers separately in your logs, so you can tell whether a failure came from finding evidence, interpreting it, or following the required format.

Compare costs using your own workload

RAG adds extraction, indexing, retrieval, and extra prompt text. Fine-tuning adds dataset preparation, training runs, model or adapter management, and regression testing. Which costs more depends on document size, update frequency, request volume, context length, model choice, and hardware. There isn't a universal rule that RAG is always cheaper or fine-tuning always faster.

For a fair comparison, use the same questions and make clear what information each version can access. An untuned model answering without the handbook is not a strong baseline for a fine-tuned model trained on its facts, just as a RAG system with stale documents isn't a fair test of retrieval. Record latency, output errors, missing answers, and the effort needed to update the system. A low average cost is less useful if the application makes expensive mistakes on the cases your users care about most.

Keep access and evidence checks outside the model

A document search service should only return passages the current user may read. Filter by permissions before placing text in a prompt, rather than asking the model to ignore confidential content it has already received. Fine-tuning introduces a different concern: examples from private records may influence later outputs, and an instruction is not a guarantee that the model won't reveal them.

Neither method removes the need to check an answer. Retrieval can supply an outdated passage, and fine-tuning can produce a confident response that doesn't fit the request. If a claim matters, compare it with its evidence or with a trusted operation that computes the result, which is what the tool-calling agent tutorial does for stock and prices. In the PDF project, Python verifies that quoted words occur in the referenced passage; it doesn't prove the answer follows from them.

Choose the next experiment you can evaluate

For the hotel-policy problem, start with the current handbook, correct version filtering, and a few questions about amounts and exceptions. For the repeated support-routing problem, start with a prompted model and held-out labelled requests, then test an adapter if a clear gap remains. If you need both specialist behaviour and current evidence, combine the two only after you can evaluate each part. Every addition should address a failure you can reproduce and a result you can check.