Fine-Tuning vs RAG vs Prompting: Which Problem Belongs to Which

The retro after a disappointing LLM feature always seems to arrive at the same sentence: “we probably need to fine-tune.” I’ve heard it from a team whose model was inventing facts, from a team whose model ignored half its instructions, and from a team whose model returned friendly prose when the parser downstream wanted JSON. Three completely different failures, one proposed fix.

The reflex is understandable. Fine-tuning sounds like real machine learning, prompting sounds like typing harder, and RAG sounds like extra infrastructure someone will have to run at 2am. So the most expensive option gets picked first because it feels like the most serious one.

 How to actually evaluate an LLM in production → Blog 10, How to Actually Evaluate an LLM in Production.

My argument is narrower than “prompting is enough” and less exciting than “fine-tune everything.” These three techniques aren’t tiers on a power ladder. They’re fixes for three different failure types: RAG fixes what the model knows, prompting fixes what the model does, and fine-tuning fixes how the model says it. Name the failure first and the choice mostly makes itself. I’m not covering pretraining your own model or reinforcement-style tuning here; that’s a different weight class with different economics, and almost nobody reading this needs it.

Key takeaways

• • The three techniques fix different failures. RAG fixes knowledge (wrong or missing facts), prompting fixes behavior (ignored instructions, wrong actions), and fine-tuning fixes form (style, structure, consistency).
  • Fine-tuning is an unreliable way to teach a model new facts. Knowledge baked into weights goes stale the day you ship and can’t cite a source.
  • Prompting has the cheapest iteration loop by far, so it’s always the first move. Most “we need fine-tuning” requests dissolve after one serious prompting pass.
  • The techniques stack because they’re orthogonal, not competing. Mature systems often run all three, each deployed on a different cadence.
  • The decision rule: ask whether the failure is knowledge, behavior, or form, and how often the underlying information changes.

Why does every LLM conversation end up at fine-tuning?

Because fine-tuning inherits the intuition of classic machine learning, where more training on your own data reliably meant a better model, and because it looks like the serious option next to editing a prompt. The reflex treats the three techniques as a ladder: prompting for prototypes, RAG for teams in the middle, fine-tuning for the grown-ups. That framing is wrong, and it’s expensive.

 [PERSONAL EXPERIENCE] Sat through the exact meeting described here at a client, where a hallucination problem was assigned a fine-tuning budget before anyone had read the failing transcripts.

I’ve sat in the meeting where a hallucination problem got assigned a fine-tuning budget before anyone had read twenty failing transcripts. The classic-ML intuition breaks on large language models for a specific reason: supervised fine-tuning on a few thousand examples mostly reshapes the distribution of outputs. It’s good at teaching the model how answers should look. It is not a dependable mechanism for storing facts you can retrieve later. Teams that fine-tune to fix hallucination usually get the same hallucinations delivered in a more confident house voice.

So what is fine-tuning actually doing, if it isn’t adding knowledge? It’s narrowing behavior. It moves the model’s defaults: tone, structure, label boundaries, formatting habits. That’s genuinely valuable for a specific class of problem, which is exactly why it survives as a technique. It’s just a much smaller class than the meeting assumes.

Where’s the line? The reflex is right when the failure genuinely is distributional: one narrow task, a large pile of clean labeled examples, and outputs that must look identical every time. That case exists, and I’ll get to it. It’s just rarer than the confidence in the room suggests.

What is prompting actually good at?

Prompting fixes behavior: which instructions the model follows, what it refuses, what shape the output takes, and how it uses the context you hand it. It’s the control plane of the whole system, and it’s the only one of the three techniques where the iteration loop is measured in minutes rather than days or weeks.

The reflex here runs the other way. Teams treat the prompt as a string literal someone wrote in April and never touched again, then conclude that “prompting doesn’t work” after three casual edits. The fix is to treat the prompt as versioned configuration that deploys like code:

# prompts/support-triage.yaml  (version-controlled, reviewed, deployed like code)
model: claude-sonnet
prompt_version: 14
system: |
  You are a support triage assistant for an infrastructure product.
  Classify each ticket into exactly one of: BUG, BILLING, HOWTO, OUTAGE.
  Respond with JSON only, in the form {"category": "...", "confidence": 0.0}
  If the ticket mentions data loss, always classify as OUTAGE.
examples:
  - input: "My invoice doubled this month with no usage change"
    output: '{"category": "BILLING", "confidence": 0.94}'

A serious prompting pass means an explicit output contract, few-shot examples chosen from real failures rather than invented ones, task decomposition when one call is doing three jobs, and stated negative rules for the mistakes you’ve actually observed. None of that needs a data pipeline, a training job, or new infrastructure. How far can instructions alone carry you? Farther than most teams ever test, because most teams iterate three times and escalate.

Two things quietly changed the economics of this layer over the last couple of years. Context windows grew to the point where “just include the whole style guide” is often a legitimate strategy rather than a hack. And prompt caching on the major APIs means a long, stable system prompt no longer costs full price on every request, which pushed the crossover point where fine-tuning wins on token cost further out than most people’s mental model says. The prompt layer got cheaper and roomier while nobody was updating their assumptions.

Where’s the line? Prompting stops paying when the fix requires information the model doesn’t have (that’s a knowledge failure, and it belongs to retrieval), when few-shot examples can’t cover the spread of your inputs, or when the prompt has grown so long that its latency and per-token cost dominate the request. A 6,000-token system prompt sent on every call is a fine-tuning candidate wearing a trench coat.

When does RAG beat fine-tuning for knowledge?

Whenever the failure is the model not knowing something: your private data, anything that changed after the training cutoff, or long-tail detail the base model only half remembers. Retrieval puts the relevant facts into the context window at request time. That single property gives you the two things weights can never give you: freshness measured in minutes, and an answer that can point at its source.

Fine-tuning for knowledge is photocopying your documents into the weights. The copy is stale the moment the original changes, it carries no provenance, and where training coverage is thin the model interpolates, which is a polite word for makes things up. Compare the update paths. New pricing page ships; with retrieval, you re-index one document:

-- freshness is a re-index, not a retraining run
SELECT id, title, chunk
FROM docs
ORDER BY embedding <=> :query_embedding
LIMIT 5;

With a fine-tuned model, the same change means rebuilding the dataset, retraining, re-evaluating, and redeploying. Which of those do you want on the critical path of a content update?

There’s also a trust dimension that gets underweighted in these debates. A retrieval-backed answer can say “according to the March runbook,” and a reviewer can check it. A weights-backed answer just sounds sure. For anything customer-facing, regulated, or auditable, provenance isn’t a nice-to-have; it’s the difference between a system you can defend and one you can only apologize for.

The honest caveat is that RAG moves the difficulty rather than removing it. The quality ceiling of a RAG system is the quality of its retrieval, which is a search problem before it’s an AI problem: chunking, hybrid lexical-plus-dense ranking, and reranking do more for answer quality than any model swap.

 Why your RAG system is actually a search problem → Blog 4, Why Your RAG System Is Actually a Search Problem.

And you likely don’t need new infrastructure to start: Postgres with pgvector covers most teams’ retrieval needs before a dedicated vector store earns its keep. 

 Most teams don’t need a vector database → Blog 2, Most Teams Don’t Need a Vector Database.

Where’s the line? RAG stops winning when the “knowledge” isn’t factual at all: tone, procedure, implicit judgment that your best people can’t write down. You can’t retrieve a writing style. And retrieval inherits search’s failure modes; if the search returns the wrong passage, generation will faithfully summarize the wrong passage.

When does fine-tuning actually pay off?

When the problem is form. Four shapes come up repeatedly: outputs that must follow a rigid structure at high volume, a house style that no prompt seems to pin down, a narrow classification or extraction task, and distilling a big model’s behavior on one task into a smaller model that’s cheaper and faster to run.

 [PERSONAL EXPERIENCE] The distillation-for-cost case reflects real client work: replacing a large-model classifier with a tuned small model on a high-volume path.

The best fine-tuning wins I’ve seen were boring ones. Not “our model learned our domain,” but “we replaced a large model plus a 2,000-token prompt with a small tuned model on a classification path that runs millions of times a month, and the latency and unit cost both dropped.” That’s the same per-call economics I walked through when comparing LLM calls to SQL queries; on a high-volume path, model size is the bill. 

 The real cost of an LLM call vs a SQL query → Blog 3, The Real Cost of an LLM Call vs a SQL Query.

The training data is the actual asset, and it looks unglamorous:

{"messages": [
  {"role": "system", "content": "Summarise the incident in our postmortem house style."},
  {"role": "user", "content": "<raw incident timeline pasted here>"},
  {"role": "assistant", "content": "Impact: ... Root cause: ... Action items: ..."}
]}

Multiply that by a few thousand clean, consistent examples and you have something worth training on. Multiply it by two hundred inconsistent ones and you have a way to make the model worse with extra steps. The other prerequisite is an evaluation set that predates the training run, because a fine-tune you can’t measure is a vibe with a GPU bill.

Worth saying plainly: the mechanics are no longer the hard part. Hosted fine-tuning on the major providers and parameter-efficient methods like LoRA on open-weight models have made the training run itself a solved, mostly affordable problem. The hard parts are the ones that were always hard: assembling a dataset your own experts agree with, and committing to re-run the whole pipeline every time the base model or the task shifts. Anyone selling you fine-tuning as a one-time purchase is selling you the first invoice.

Where’s the line? Fine-tuning stops paying the moment the underlying information changes faster than your retraining cadence, when you have hundreds of examples rather than thousands, or when the provider’s next base model erases your gain for free. Weights are the slowest-moving layer in the stack. Only slow-moving things belong in them.

Can you stack all three together?

Yes, and mature systems usually do, because the techniques are orthogonal rather than competing. A production assistant might use retrieval for facts, a versioned prompt for behavior and the output contract, and a fine-tuned small model for the high-volume classification step that routes requests in front of everything else. Each one is doing the only job it’s good at.

 [UNIQUE INSIGHT] The deploy-cadence framing (prompt per commit, index per document, weights per quarter) as the test for where a fix belongs.

The framing I keep coming back to: these are three configuration layers with three deploy cadences. The prompt changes per commit. The index changes per document. The weights change per quarter, if that. The cadence test also tells you where any given fix belongs. If you’d want to change something weekly, it must not live in the weights. If it must be true for every request regardless of context, it can’t rely on retrieval alone.

The order of operations follows from cost of iteration, not from prestige. Prompt first, because the loop is hours. Retrieval second, because the loop is days. Fine-tune last, because the loop is weeks and it’s the only one that needs a labeled dataset to even start. Each escalation should happen only after the previous layer measurably failed on cases you can point to, not after it merely felt insufficient.

A concrete shape, since abstractions hide the point: a support assistant where a fine-tuned small classifier routes tickets (form problem, millions of calls, stable labels), retrieval pulls the product docs and the customer’s plan details (knowledge problem, changes weekly), and a versioned prompt enforces tone, escalation rules, and the JSON reply contract (behavior problem, changes with every product decision). Remove any one layer and a different failure class comes back. Swap two of them and you’ve built the expensive version of the wrong fix.

Where’s the line? Stacking stops helping when you can no longer attribute a failure to a layer. Without per-layer checks (retrieval hit rate, instruction-following spot checks, format accuracy), the layers hide each other’s faults and every incident turns into archaeology. If you can’t say which layer broke, you’ve stacked past your observability.

So how do you actually decide?

Four questions, asked in order, settle nearly every case I’ve come across.

1. What kind of failure is it? Read the failing transcripts and classify. Wrong or missing facts is a knowledge failure: retrieval. Ignored instructions, wrong tool choices, bad refusals is a behavior failure: prompting. Right content in the wrong shape, tone, or level of consistency is a form failure: fine-tuning. Most teams skip this step, which is how a knowledge failure ends up with a training budget.

2. How often does the underlying information change? Daily or weekly means it lives in context, through retrieval or the prompt. Quarterly or never means it’s at least eligible for weights. Nothing that changes monthly should ever be trained in.

3. Can you write the fix down as an instruction? If a competent new hire could apply the fix from a written note, it’s a prompt change. If they’d need to consult the docs each time, it’s retrieval. If they’d need six months of imitating a senior colleague to absorb it, that’s the kind of implicit pattern fine-tuning can capture.

4. Do you have the examples and the eval to justify training? No golden set means you’re not ready to fine-tune, whatever the answer to the first three questions. Build the evaluation first; it’s the same asset you need to run any of this responsibly. 

 Building a golden set from production traffic → Blog 10, How to Actually Evaluate an LLM in Production.

Read the answers honestly and most requests land on prompting or retrieval. Fine-tuning is the correct answer to a real but small set of problems, and when it’s right, it’s very right: the boring, high-volume, stable-form cases where it quietly prints money. 

 LLM cost optimisation, caching, routing, model selection → Blog 22, LLM Cost Optimisation: Caching, Routing, Model Selection.

Match the failure, then pick the tool

The practical default is unexciting. Prompt until it measurably fails, with the prompt under version control and real failures driving each revision. Add retrieval when the failures are factual, and accept that you’ve just taken on a search-quality problem. Fine-tune when the failures are about form, the underlying task is stable, and you have thousands of examples plus an eval that predates the run.

Get the matching wrong and the costs aren’t symmetric. Over-prompting wastes an afternoon. Premature RAG wastes a sprint on infrastructure you may not need yet. Premature fine-tuning burns weeks and a labeling budget to ship a model that repeats stale facts in a beautifully consistent voice, and when someone asks where an answer came from, nobody can say.

Fine-tuning vs RAG vs prompting was never a ranking. It’s a matching problem, and the match is decided by the failure, not by the tool’s prestige. Name which problem you actually have, and it’ll tell you which technique it belongs to.

Frequently asked questions

• Is fine-tuning better than RAG for accuracy?

No, they target different failures. RAG improves factual accuracy because it grounds answers in retrieved sources that stay current and can be cited. Fine-tuning improves consistency of style and structure. Fine-tuning a model to fix hallucination usually produces the same hallucinations delivered in a more polished and confident voice.

• Can fine-tuning teach an LLM new facts?

Only unreliably. Fine-tuning mainly reshapes how a model responds rather than storing knowledge you can retrieve on demand. Facts trained into weights go stale immediately, carry no source, and blur together where training coverage is thin. If the failure is missing or outdated information, retrieval is the dependable fix.

• Should I always try prompting before RAG or fine-tuning?

Yes. Prompting has the fastest iteration loop and zero infrastructure cost, so it establishes the baseline every other technique must beat. Write an explicit output contract, add few-shot examples taken from real failures, and measure the result. Most requests for fine-tuning dissolve during that one serious pass.

• When is fine-tuning worth the cost?

When outputs must follow a strict form at high volume, when a smaller fine-tuned model can replace a larger one to cut latency and per-request cost, or when a narrow classification task has thousands of clean labeled examples. You also need an evaluation set built before training, to prove the gain.

• Do I need a vector database for RAG?

Usually not. Postgres with pgvector handles retrieval well at the scale most teams operate, and it keeps embeddings next to the data they describe. A dedicated vector database earns its place at very large corpus sizes or strict latency floors, not as the default starting point for a first syste