Prompting is the right default: it is instant, cheap, and reversible. Reach for retrieval (RAG) when the model needs facts it was never trained on. Fine-tune only when you need a consistent behavior — a fixed format, tone, or classification boundary — that prompting alone cannot hold reliably at scale. Most teams should prompt first, add retrieval second, and fine-tune last.
That is the short version. The rest of this guide is the decision framework behind it: what each approach actually changes inside the model, the questions that separate a prompting problem from a retrieval problem from a training problem, and the cost and maintenance math that decides the case at real volume.
Fine-tuning vs prompting: what each one actually changes
The confusion starts because all three techniques look like “make the AI better.” They act on completely different parts of the system, and that difference is what makes one of them correct and the other two wasteful for any given problem.
- Prompting changes the instructions for one call. Nothing persists. You steer behavior with system prompts, few-shot examples, and explicit output rules. Zero training, zero infrastructure.
- Retrieval (RAG) changes the context for one call. You fetch relevant documents from your own corpus and paste them into the prompt. The model’s weights never move; you are just handing it better notes.
- Fine-tuning changes the weights. You run additional training on examples so the behavior is baked into the model itself. It persists across every call, costs money up front, and creates a new artifact you must version and maintain.
Read that list again and the strategic implication falls out immediately: prompting and retrieval are runtime decisions you can change in a deploy, while fine-tuning is a build decision you live with for months. That asymmetry is why the bar for fine-tuning should be much higher than the bar for a new prompt.
The one question that resolves most cases
Ask: is my problem about knowledge, or about behavior?
If the model fails because it does not know something — your product docs, last quarter’s policies, a customer’s account history — that is a knowledge gap, and retrieval fixes it. Fine-tuning on facts is the classic expensive mistake: the facts go stale, you have to retrain, and the model still hallucinates them because trained-in knowledge is not verifiable.
If the model knows what it needs to know but keeps behaving wrong — ignoring your JSON schema, drifting out of tone, mis-classifying edge cases, over-explaining when you asked for one line — that is a behavior gap, and fine-tuning is a genuine candidate. It is also the only case where training usually pays for itself.
| Your symptom | Likely gap | Right first move |
|---|---|---|
| Model doesn’t know our internal docs | Knowledge | Retrieval (RAG) |
| Model knows the facts but formats output wrong | Behavior | Prompting, then structured outputs |
| Answers are outdated after a policy change | Knowledge | Retrieval |
| Output schema breaks 5–10% of the time | Behavior | Prompting + schema enforcement, then fine-tune if it persists |
| Tone is inconsistent across thousands of calls | Behavior | Few-shot prompting, then fine-tune |
| Classification accuracy plateaus below target | Behavior | Fine-tune on labeled examples |
| Task needs long, stable reasoning style | Behavior | Fine-tune or distillation |
| Latency too high from a huge prompt | Both | Fine-tune to shrink the prompt |
When prompting is the answer (and it usually is)
Prompting wins whenever the task is expressible in words and the model already has the underlying capability. That covers a surprising amount of production work: drafting, summarizing, rewriting, extracting, classifying, and most conversational flows.
The reason to start here is not just cost. It is iteration speed. A prompt change ships in seconds and rolls back in seconds. A fine-tune takes a data-collection cycle, a training run, an evaluation pass, and a deployment — days to weeks per iteration. If you fine-tune before you have exhausted prompting, you have made your slowest possible loop your only loop.
Two prompting upgrades deserve to be tried before you consider training at all. The first is few-shot examples: five to ten well-chosen input/output pairs often close most of the quality gap that people assume requires a fine-tune. The second is enforced structure — if your real complaint is malformed JSON, the fix is a structured output mode, not a training run. Our guide to structured outputs and JSON mode walks through schema enforcement and why it removes the single most common reason teams reach for fine-tuning too early.
When retrieval (RAG) is the answer
Choose RAG whenever the correct answer depends on information that changes, is private, or is too large to fit in a prompt. Support knowledge bases, product documentation, legal and policy text, and customer-specific data are all retrieval problems.
RAG has three properties that make it strictly better than fine-tuning for knowledge work:
- Freshness. Update a document and the next request sees it. No retraining, no deployment.
- Attribution. You can cite which chunk produced the answer, which is non-negotiable for compliance and for user trust.
- Access control. Permissions live in your retrieval layer, so one user never sees another’s documents. A fine-tune cannot do this — once facts are in the weights, every caller gets them.
The mechanics are covered in depth in our embeddings and RAG guide: chunking strategy, embedding model choice, hybrid search, and re-ranking. The short version is that retrieval quality dominates answer quality, so spend your effort on chunking and re-ranking before you spend a dollar on training.
When fine-tuning is genuinely worth it
Fine-tuning earns its cost in four situations. If none of them describe you, keep prompting.
- Behavior that must be identical every time. Strict output contracts, regulated language, brand voice at scale. You need determinism that prompt engineering only approximates.
- A narrow task at high volume. Fine-tuning a small model to match a large model on one specific task (distillation) can cut per-request cost by an order of magnitude — but only if the volume is there to amortize the training run.
- Prompt bloat. If your system prompt has grown to thousands of tokens of rules and examples, you are paying that cost on every call. A fine-tune can compress it into the weights and cut both latency and input tokens.
- A quality ceiling. When you have measured that a well-crafted prompt plateaus below your accuracy target and you have labeled data to fix it, training is the honest next step.
LoRA and parameter-efficient fine-tuning
Modern fine-tuning rarely means retraining the whole model. LoRA (Low-Rank Adaptation) freezes the base weights and trains small adapter matrices instead. The practical consequences are what matter for a decision:
- Far cheaper to train. You are optimizing a fraction of the parameters, so runs are shorter and can often fit on a single GPU.
- Small artifacts. Adapters are megabytes, not gigabytes, so versioning and swapping them is easy.
- Composable. One base model can serve several adapters — a support-tone adapter and a legal-tone adapter — selected per request.
- Still not free. You now own a dataset, an adapter registry, an eval harness, and a re-training schedule when the base model is upgraded.
That last point is the one teams underestimate. A fine-tune is not a one-time purchase; it is a subscription to maintenance. Budget for it explicitly, or you will find yourself pinned to an old base model because nobody wants to redo the training run.
A decision procedure you can code
The framework above compresses into a small routing function. In practice you would gate this on measured evaluation scores rather than booleans, but the shape is the same:
def choose_technique(task):
"""Pick the cheapest technique that can actually close the gap."""
# 1) Is the failure about missing or changing facts?
if task.needs_private_data or task.data_changes_frequently:
return "RAG" # retrieval layer, no training
# 2) Is the failure about format or contract compliance?
if task.requires_schema:
if task.schema_violation_rate < 0.02:
return "prompt + structured outputs"
# still failing after schema enforcement + few-shot?
return "fine-tune (LoRA) on labeled examples"
# 3) Is the failure about tone / behavior consistency at volume?
if task.consistency_score < task.quality_bar:
if task.examples_labeled >= 500 and task.volume_per_month > 100_000:
return "fine-tune (LoRA)"
return "few-shot prompting" # iterate here first
# 4) Is the failure actually about cost or latency?
if task.prompt_tokens > 2000 and task.volume_per_month > 100_000:
return "fine-tune to compress the prompt"
return "prompting" # the correct default
Two thresholds in that snippet carry most of the weight. The labeled-example count is a floor: below a few hundred high-quality examples, fine-tuning overfits and you learn nothing reliable. The volume number is the amortization test — training cost divided by monthly requests has to be small enough that the efficiency gain wins. Run the arithmetic before you run the training job.
The cost and effort math, in ratios
Exact prices move constantly, so reason in relative terms. The stable picture looks like this:
| Dimension | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| Up-front effort | Minutes | Days | Weeks |
| Iteration loop | Seconds | Hours | Days |
| Per-request cost | Baseline | Higher (retrieved context) | Lower (shorter prompt, smaller model) |
| Data needed | A few examples | A document corpus | Hundreds to thousands of labeled pairs |
| Handles fresh facts | No | Yes | No |
| Reversible | Instantly | Instantly | Only by retraining |
| Ongoing maintenance | Prompt edits | Index refresh | Adapter + eval + base-model upgrades |
The pattern to notice is that fine-tuning is the only column with a negative on reversibility. Everything else you can undo with a deploy. That single row is why the correct ordering is almost always prompt → retrieve → train, and why the training step should be justified by measurement rather than by frustration.
Combining them: the production pattern
These are not mutually exclusive, and mature systems use all three at once. A common production shape: retrieve relevant context with RAG, send it to a fine-tuned small model that has learned your exact output contract, and keep a frontier model in the fallback chain for requests the small model scores as low-confidence.
That combination only works if you can move between models freely. If every provider has a different base URL, auth scheme, and request shape, then “try a fine-tuned small model and fall back to frontier” turns into a refactor instead of a config change. A unified, OpenAI-compatible endpoint collapses that to a single string. That is the problem an AI API relay solves, and it is why the model-selection layer should be decoupled from the technique layer: you want to be able to swap the model underneath a fine-tuned workflow without rewriting anything. Our guide to choosing the right AI model and routing requests covers the tiering and fallback design in detail, and qoraapi.com exposes many models through one such endpoint if you want to test the pattern without wiring up four vendor accounts.
Common mistakes
- Fine-tuning to add facts. The most expensive way to build a worse search index. Use retrieval.
- Skipping few-shot prompting. Teams frequently spend a training budget solving a problem that ten good examples would have solved.
- Training on a dirty dataset. Your model learns your labeling errors, faithfully and at scale. Audit the data before you train.
- No eval harness. Without a frozen test set you cannot tell whether the fine-tune helped or just changed the failure mode. Build the eval before the dataset.
- Ignoring the maintenance bill. Every base-model upgrade forces a decision about re-training. Plan for it.
- Assuming structure requires training. Malformed JSON is a decoding problem, not a weights problem.
Frequently asked questions
Is fine-tuning better than prompting?
Not in general — they solve different problems. Prompting changes instructions for one call; fine-tuning changes the model’s weights permanently. Fine-tuning is better only when you need consistent behavior that prompting cannot hold, you have hundreds of labeled examples, and your volume amortizes the training and maintenance cost. For knowledge gaps, retrieval beats both.
Can fine-tuning replace RAG?
Rarely, and it is usually the wrong trade. Fine-tuning cannot guarantee factual accuracy, cannot cite sources, cannot enforce per-user access control, and goes stale the moment your documents change. Use RAG for knowledge and fine-tuning for behavior; when you need both, run them together — retrieve first, then pass the context to a fine-tuned model.
How much data do I need to fine-tune a model?
For narrow behavior shaping, a few hundred high-quality input/output pairs can be enough with parameter-efficient methods like LoRA. Below that, few-shot prompting is more reliable. The binding constraint is usually quality, not quantity: a thousand clean, consistent examples beat ten thousand noisy ones every time.
What is the difference between RAG and fine-tuning?
RAG retrieves relevant text at request time and puts it in the prompt, leaving the model unchanged. Fine-tuning modifies the model’s parameters during a training run. RAG is fresh, attributable, and instantly reversible; fine-tuning is persistent, lower-latency at inference, and costly to change. RAG answers “what does the model need to know right now,” fine-tuning answers “how should the model always behave.”
Does fine-tuning reduce cost?
It can, but not automatically. The savings come from two places: replacing a large prompt with learned behavior, and distilling a frontier model’s skill on one narrow task into a small model. Both require volume to pay back the training run. If your request volume is modest, prompting plus retrieval will be cheaper overall.
Conclusion
Prompt first, because it is fast and reversible. Add retrieval when the gap is knowledge, not capability. Fine-tune only when you have proven — with an eval set, not a feeling — that a consistent behavior is out of reach for prompting, and when your volume justifies the training and maintenance cost. Getting that order right is worth more than any single technique, because it keeps your iteration loop measured in seconds for as long as possible.
If you want to experiment with the model layer without changing providers, start from our AI API gateway guide and the OpenAI-compatible API explainer, then apply the decision procedure above to your own task.
Leave a Reply