Fine-Tuning, LoRA and When Not to Fine-Tune

Teach style and format, not facts

Fine-tuning continues training a pretrained model on your own examples. Parameter-efficient methods such as LoRA freeze the original weights and train small added matrices, cutting memory and cost dramatically and producing small adapter files. Fine-tuning is good for a consistent style, format, tone or narrow skill. It is a poor way to add fast-changing knowledge: use retrieval (RAG) for facts that change or must be cited, and try better prompts and examples first, because they are cheapest. A common order of attempts is prompt, then retrieval, then fine-tuning.

Start with 50 great examples

Quality beats quantity: a few dozen carefully written examples often show whether fine-tuning will help before you invest in thousands.

Quick check: Which problem is best solved with retrieval rather than fine-tuning?

• Answering from documents that change daily and must be cited
• Always replying in a fixed brand tone
• Learning a fixed output format
• Making the model smaller