# Fine-Tuning, LoRA and When Not to Fine-Tune — Large Language Models

Source: https://www.geekswithgeeks.com/en/llms/r-finetune

> Adapt a model cheaply and choose between prompting, retrieval and tuning.

## Teach style and format, not facts

**Fine-tuning** continues training a pretrained model on your own examples. **Parameter-efficient methods** such as **LoRA** freeze the original weights and train small added matrices, cutting memory and cost dramatically and producing small adapter files. Fine-tuning is good for a consistent **style, format, tone or narrow skill**. It is a poor way to add fast-changing knowledge: use **retrieval** (RAG) for facts that change or must be cited, and try **better prompts and examples** first, because they are cheapest. A common order of attempts is prompt, then retrieval, then fine-tuning.

## Start with 50 great examples

Quality beats quantity: a few dozen carefully written examples often show whether fine-tuning will help before you invest in thousands.

**Quiz:** Which problem is best solved with retrieval rather than fine-tuning?

- [x] Answering from documents that change daily and must be cited
- [ ] Always replying in a fixed brand tone
- [ ] Learning a fixed output format
- [ ] Making the model smaller

*Answer:* Answering from documents that change daily and must be cited. Retrieval brings current, citable facts into the prompt at question time.