Should you fine-tune a small open model on your own data, use RAG, or just write a better prompt?

Fine-tuning changes how a model behaves. It does not give it a filing cabinet. Train a model on your product manual and it will sound like your product manual and still get the part numbers wrong, because weights store patterns, not records. That is the single misunderstanding behind most failed fine-tunes, and Unsloth's own guide says it plainly: facts that change or need citing belong in RAG, which looks the right passage up and puts it into the prompt at question time.

Fine-tuning earns its cost for three things. Tone and format, when every answer must be in your house style or a strict JSON shape and the prompt keeps drifting. A narrow repeated task that a small model does badly and a big one does well, where you have hundreds of graded examples. And offline or small-footprint use, where you cannot carry a 70B model and a long system prompt around. Before either, there is the cheapest option of all: a better prompt with five to ten examples in it. It solves more problems than both and costs ten minutes.

Often the answer is both: RAG for the facts, a fine-tune for the voice. Press through the tree with your actual case, then do step 2 whatever it said, because step 2 builds the test set the rest of the guide needs.

Whatever the tree said, do this step, because it builds the evaluation you need later. Write a system prompt that states the task, the format and the rules, then paste in five to ten real examples of input and ideal output. Run twenty fresh inputs through it on the model you plan to use, the small open one, not a cloud one. Grade each answer pass or fail against what you would have accepted from a person.

If sixteen or more pass, you are done and you saved a week. If the failures are factual, wrong dates, wrong names, invented policies, the fix is RAG and no amount of training will help. If the failures are stylistic, wrong format, too long, ignores a rule, wrong register, you have a fine-tuning case, and the twenty graded inputs are now the first rows of your test set. Keep them out of the training data for good. A model that has seen the test is a model you cannot measure.

The prompt below is the shape we use. Replace the examples with yours and keep them short; the model copies length as readily as it copies style.

Unsloth publishes the numbers and they are the ones we size against. For a QLoRA run, which trains small adapter layers on top of 4-bit weights, a 3B model needs about 3.5 GB, an 8B 6 GB, a 14B 8.5 GB, a 27B 22 GB and a 70B 41 GB. For 16-bit LoRA, where the base model sits in memory in full, those become 8, 22, 33, 64 and 164 GB. The gap is why nearly everyone fine-tunes with QLoRA; the quality difference is small and the memory difference is a different class of hardware.

A free Google Colab notebook gives you a T4 with 16 GB. That fits 8B QLoRA, which is the sweet spot for most teams: big enough to learn a format, small enough to run on a laptop afterwards. Sequence length raises memory beyond the table, and Unsloth does not publish a per-token figure, so above 2,048 tokens leave headroom or drop a size.

The training time band in the calculator is our rough rule from our own runs, not a benchmark, and it can be off by a factor of two in either direction. A free Colab session can end without warning, so anything that reads as hours belongs on a rented GPU or on a machine you own.

Unsloth is the training library we use because it is the fastest path from a Hugging Face model to a GGUF file Ollama can run, and because its requirements are plain: Linux, Windows or WSL, an NVIDIA GPU with CUDA 12.4 or newer, and Python 3.11 to 3.13. Install it with pip into a fresh environment. On a free Colab notebook the runtime already has CUDA; pick the T4 GPU runtime and run the same pip line in the first cell.

Start from an Instruct model, not a base one. An Instruct model already knows how to hold a conversation, so your few hundred examples only have to teach the specifics. Unsloth republishes popular models pre-quantised under names ending in -unsloth-bnb-4bit; they download faster and skip a conversion step. Pick a current one from Unsloth's page on Hugging Face and check the exact name there, because the names below are the pattern, not a promise.

Load it with FastLanguageModel.from_pretrained, set max_seq_length to 2048 to match the calculator, and let load_in_4bit=True do the QLoRA part. Then get_peft_model adds the adapters; r=16 and lora_alpha=16 are the notebook defaults and we have not needed to change them for a format task. Run the lines below before you touch your data. If the model loads and answers one question, the hardware is settled.

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-8B-unsloth-bnb-4bit",  # check the current name
    max_seq_length=2048, load_in_4bit=True)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"])

The data is the whole fine-tune; the code is boilerplate. You need pairs: the input a person would give and the output you want, in the chat format the model was trained on. For Instruct models that means a conversation with system, user and assistant turns, which Unsloth accepts in ShareGPT or ChatML style and converts with the model's own chat template, so the model trains on exactly the tokens it will see later. One JSON object per line.

What goes in the assistant turn is what the model learns to produce. If your answers contain facts, it learns the facts as they were on the day, and it learns to state facts confidently, including wrong ones. So strip the volatile facts out of the answers, or move them into the user turn as context, exactly as RAG would supply them at run time. Teach the shape, not the content.

Three hundred clean, consistent examples beat three thousand scraped ones; the model copies your inconsistencies with the same enthusiasm as your style. Hold out the twenty graded cases from step 2 and another ten percent at random. Print one row after the template has been applied and read the raw tokens once. That single look catches most template bugs before they cost a training run.

{"conversations": [
  {"role": "system", "content": "You write replies to customer emails. Under 120 words, one next step, never promise a date."},
  {"role": "user", "content": "Context: order 4821 shipped 12 Oct, carrier shows held at customs.\n\nEmail: Hi, my order still has not arrived and nobody answers."},
  {"role": "assistant", "content": "Hi Dana,\n\nSorry for the wait. Order 4821 is being held at customs ... \n\nSam"}
]}

Training is one object and one call. SFTTrainer from TRL takes the model, the tokenizer, the dataset and a config. The settings that matter, from Unsloth's guide: learning_rate=2e-4, which is high by pre-training standards and right for LoRA; num_train_epochs between 1 and 3, and we start at 2; gradient_accumulation_steps=4 with a batch size of 2, which is why the calculator counts eight rows per step; max_seq_length=2048. The exact argument name for the sequence length has moved between TRL versions, so copy it from the current Unsloth notebook. Everything else can stay at the notebook defaults on the first run.

Watch the training loss. It should fall quickly in the first fifty steps and then flatten. If it keeps sliding towards zero, the model is memorising; that is the overfitting trap on a few hundred rows at three or more epochs, and the fix is fewer epochs, not more data. If it never falls, the chat template is wrong and the model is being trained on the wrong tokens; go back to step 5.

Save the adapter with model.save_pretrained at the end. It is a few hundred megabytes and it is the only thing you need to keep; the base model is downloaded again. Then grade the held-out rows with the same pass or fail test as step 2, before anything is exported.

from trl import SFTTrainer, SFTConfig
trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=dataset,
    args=SFTConfig(learning_rate=2e-4, num_train_epochs=2,
        per_device_train_batch_size=2, gradient_accumulation_steps=4,
        max_seq_length=2048, output_dir="out"))
trainer.train()
model.save_pretrained("lora-adapter"); tokenizer.save_pretrained("lora-adapter")

Test in the same process before you export. FastLanguageModel.for_inference(model) switches to fast generation; build the prompt with tokenizer.apply_chat_template, the same template training used, and generate. Run the held-out cases through the base model and the tuned one on the same inputs, side by side. The point is not whether the answers are good. It is whether they beat the step 2 prompt on the same small model. If they do not, the fine-tune has not paid for itself, and that is a fine thing to learn before the export dance.

Look for three failures. Answers that quote training rows word for word: overfitting, drop an epoch. Answers that are right in shape but wrong in fact: the facts crept into the assistant turns, or the user turn had no context, and the fix is RAG at run time. Answers that run on, repeat, or never end: the end-of-sequence token did not make it into training, which happens when the data was formatted without the template's closing token. Fix that here, because after the GGUF export the same problem looks like a different bug and you will chase it in the wrong place.

Write the pass rate down. It is the number step 9 checks against.

Ollama runs GGUF files, and Unsloth writes them in one call. q4_k_m is the size most people want; q8_0 is twice the size and a little closer to the original; f16 is the full thing for a GPU with room. The call merges the adapter into the base weights, converts, and writes a Modelfile into the same folder, with the chat template and stop tokens filled in from the tokenizer. That Modelfile is the part people lose. Keep it next to the adapter, in version control.

Then two Ollama commands: create a model from the Modelfile, and run it. If the export dies with an out-of-memory error, lower maximum_memory_usage in the save call from the default 0.75 to 0.5; the merge step needs the full-precision weights for a moment. On Colab the GGUF is a multi-gigabyte file, so download it or push it to Hugging Face with push_to_hub_gguf rather than leaving it in a session that will expire with it inside.

model.save_pretrained_gguf("email-model", tokenizer, quantization_method="q4_k_m")
# out of memory on the merge: add maximum_memory_usage=0.5

The last mile is where fine-tunes go to die quietly. In Ollama the Modelfile's TEMPLATE and its stop PARAMETER lines must match the chat template and end-of-sequence token used in training. When they do not, the model is fed tokens it never saw, and the result is gibberish, the wrong language, or an answer that never stops. Unsloth's Modelfile gets this right for the models it knows. It is still worth reading once with ollama show, and comparing with what tokenizer.chat_template printed in step 7.

There is also a known silent failure. Unsloth issue #5070 (16 April 2026) reports that the Gemma 3 GGUF export writes special tokens as NORMAL instead of CONTROL, and chat inference breaks without an error. If you export a Gemma model and the output looks fine in Python and wrong in Ollama, read that issue before you touch your data.

Then run the held-out cases a third time, now through ollama run, and grade them. The same pass rate as step 7 means the export is clean, and the model is ready for the team or for the Open WebUI setup in our self-hosting guide. If the rate dropped, pick the symptom below.