Fine-Tune an LLM with LoRA and QLoRA: A Practical Python Tutorial

Suppose your support team needs each incoming message labelled as shipping, refund, or account. A language model can often do that with a good prompt, so try that first, for example on a local model through Ollama. If the prompt still gives inconsistent labels across many requests, it's time to fine-tune an LLM on examples. LoRA lets you do that without updating every weight in the original model, which keeps the experiment small and the result easy to store.

We'll use a deliberately small routing task to see what changes during training and what can still go wrong afterwards. The runnable example uses DistilGPT2, an 82-million-parameter model. That's tiny by LLM standards: a stand-in that trains on a laptop CPU in minutes, not a model to put behind a real support queue. The same script accepts --model, so you can swap in a larger instruct model once the mechanics make sense. Download the projects or open lora_train.py. We'll run ordinary LoRA first, then look at the QLoRA configuration for a compatible CUDA machine.

How LoRA fine-tuning changes a layer

An ordinary linear layer multiplies its input by a weight matrix, $W$. Full fine-tuning updates that matrix, along with the other trainable weights. LoRA instead freezes the original matrix and learns a smaller update through two matrices, $A$ and $B$. The layer uses $W + sBA$, where the scaling factor is $s = \alpha / r$: the lora_alpha setting divided by the rank. With the configuration we use below, lora_alpha=16 and r=8, so $s = 2$. In the LoRA paper, this low-rank parameterization is the central idea.

If $W$ has 4,096 rows and 4,096 columns, it contains 16,777,216 values. With rank eight, the two update matrices together contain 65,536 values: eight times 4,096 in each matrix. That's the parameter count for one simplified layer, not the memory footprint of a training run. The frozen model still has to be loaded, and activations, adapter gradients, and optimizer state need memory too.

rows = columns = 4096
rank = 8
full_parameters = rows * columns
adapter_parameters = rank * columns + rows * rank
print(full_parameters)     # 16777216
print(adapter_parameters)  # 65536

How QLoRA differs

QLoRA keeps the same idea of training adapters while storing the frozen base weights in a lower-precision representation. The QLoRA paper introduces four-bit NormalFloat quantization, double quantization, and paged optimizers. Our example uses the first two through bitsandbytes, but keeps a normal AdamW optimizer. It's a QLoRA-style configuration, not a reproduction of the paper's experiments on much larger models.

Quantizing the frozen weights doesn't make every number in training four bits. The compute dtype, adapter parameters, optimizer state, and activations all matter. On the CUDA path, the code chooses bfloat16 compute when supported and otherwise float16. The Transformers bitsandbytes guide lists supported backends. This script uses the NVIDIA CUDA route for QLoRA only; it doesn't claim to work on the Mac's MPS backend.

Prepare a small training task

The starter has twelve fictional support requests with one label each, and three separate held-out requests. That's enough to run the mechanics and see a failure, but nowhere near enough to estimate how a real support router would perform. A real dataset needs representative examples, ambiguous cases, a consistent labelling policy, and a held-out set that isn't a lightly reworded copy of the training messages. Keep customer identities and private conversations out of it.

TRAIN = [
    ("Where is my parcel?", "shipping"),
    ("Please refund this purchase.", "refund"),
    ("I cannot log in.", "account"),
    # The downloadable file contains all twelve examples.
]
HELD_OUT = [
    ("Has my package been dispatched?", "shipping"),
    ("Can I get a reimbursement?", "refund"),
    ("I forgot my sign-in password.", "account"),
]

Install the training dependencies in their own environment. Install PyTorch for your platform first, using the selector on pytorch.org, then the pinned packages from requirements-lora.txt. The commands below run the CPU path with 24 optimizer steps, matching the recorded run. On an Apple Silicon Mac, the script also accepts --device mps if your PyTorch build supports it; we haven't run that path. The base model and tokenizer download the first time you run the script.

unzip ai-blog-starters.zip
cd ai-blog-starters
python3 -m venv .venv-lora
source .venv-lora/bin/activate
python -m pip install torch  # or the command pytorch.org gives for your platform
python -m pip install -r requirements-lora.txt
python lora_train.py --device cpu --steps 24 --output routing-adapter
# Optional on a supported Apple Silicon setup:
# python lora_train.py --device mps --steps 24 --output routing-adapter

To try a more realistic model, pass a Hugging Face identifier such as a half-billion-parameter instruct model: python lora_train.py --model Qwen/Qwen2.5-0.5B-Instruct --device cpu --steps 24 --output routing-adapter-qwen. Expect a bigger download and slower steps. We haven't recorded results for that model, so compare its held-out predictions yourself.

Attach LoRA adapters with PEFT

The script loads the tokenizer and pretrained base model before wrapping it with PEFT's get_peft_model. We use rank eight, alpha sixteen, dropout 0.05, and target_modules="all-linear". These are example settings, not values tuned for support routing. The PEFT quicktour describes this wrapping and saving workflow. Print the trainable parameters: you want to see the adapters as trainable and the original weights frozen.

from peft import LoraConfig, get_peft_model
 
config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

The target modules depend on the architecture and on which parts of the model you want to adapt. all-linear avoids hard-coding names from a different model, but it doesn't tell you which layers give the best tradeoff for your task. A higher rank allows a larger update and adds trainable parameters; it doesn't guarantee better answers. Record rank, learning rate, data, and evaluation results together so you can compare runs.

Train on the answer tokens

Each example becomes a prompt such as Request: Where is my parcel? followed by Intent: and the expected label. The code tokenizes the prompt and completion separately, joins them, and sets prompt positions in the label tensor to -100. The loss function ignores those positions, so training focuses on the completion instead of rewarding the model for copying the request. The attention mask still includes the prompt, because the model needs to read the request to predict its label.

prefix = tokenizer.encode(prompt(text), add_special_tokens=False)
suffix = tokenizer.encode(" " + label, add_special_tokens=False)
suffix += [tokenizer.eos_token_id]
ids = torch.tensor([prefix + suffix], device=args.device)
labels = ids.clone()
labels[:, :len(prefix)] = -100
 
optimizer.zero_grad()
loss = model(
    input_ids=ids,
    attention_mask=torch.ones_like(ids),
    labels=labels,
).loss
loss.backward()
optimizer.step()

The full script sets the random seed, trains one short example per step, and records predictions on the three held-out requests before and after training. It also checks that the frozen base parameters stayed unchanged. One-example batches keep the loop easy to follow. A real recipe would need proper batching, sampling, and sequence-length handling.

Read the result, including the mistake

In the recorded 24-step CPU run, training loss fell from about 8.78 to 0.34 and the frozen base weights stayed unchanged. Two held-out requests received the expected labels. The shipping question “Has my package been dispatched?” was labelled refund.

That mistake is the useful part. The model learned to emit the label format, but lower training loss didn't mean it understood the routing task. The training report keeps the predictions and settings, including the failed example.

Evaluate formatting and routing separately. A response with exactly one allowed label meets the output contract but can still send a ticket to the wrong team. Add enough held-out requests to cover the cases you care about, compare against the untuned model with a sensible prompt, and include ambiguous messages and requests outside the three labels. If the adapter only succeeds on near-copies of the training set, more optimizer steps won't fix that.

Save and reload the adapter

save_pretrained stores the trained adapter and its configuration, while the base model remains a separate dependency. The script also saves the tokenizer and a JSON training report in the output folder. Keep the base model identifier, and for reproducible work its revision, with the adapter: loading it onto a different architecture or base revision isn't the model you trained.

model.save_pretrained("routing-adapter")
tokenizer.save_pretrained("routing-adapter")

To classify a new request with the saved adapter, run lora_predict.py with the adapter folder and the request text. It restores the base model from the saved configuration, applies the adapter, and prints the generated label. In the recorded run, this request was labelled refund.

python lora_predict.py routing-adapter "Can I get a reimbursement?"

To try the QLoRA path, use a supported CUDA environment and install bitsandbytes there. The script loads the base with NF4 four-bit quantization, enables double quantization, and calls prepare_model_for_kbit_training before attaching adapters, following the PEFT quantization guide. We ran the CPU LoRA path; the GPU QLoRA path hasn't been run, so we make no claim about its runtime or memory on your machine.

python -m pip install bitsandbytes
python lora_train.py --qlora --device cuda --steps 24 --output routing-qlora

Decide whether training actually helped

A useful adapter improves an agreed set of tasks without creating unacceptable failures elsewhere. For this example, that means correct routing on new messages, predictable output formatting, and a clear way to handle requests outside the supported categories. Collect those results before deciding that a smaller training loss is good enough.

If your real problem is that the model doesn't know a changing policy document, read RAG versus fine-tuning before putting those facts into training examples. Retrieval may address the missing information more directly. The RAG evaluation guide shows how to build the kind of held-out test set this article keeps asking for.