Fine-Tuning vs RAG vs Prompt Engineering: Which Does Your Use Case Need?

Most teams that ask us about fine-tuning don’t need it. Their model gets facts wrong or answers in the wrong shape, and training a custom model sounds like the grown-up fix. Usually a better prompt or a retrieval layer does the job.

Here is the short version of fine tuning vs RAG vs prompt engineering. Prompt engineering is always step one. RAG fixes missing or changing knowledge. Fine-tuning fixes behavior the model can’t learn from instructions, such as a strict output format, a narrow classification task, or a smaller model doing a bigger model’s job. And in November 2026 there is a new factor most comparison articles haven’t caught up with: the supply of fine-tuning has shrunk. OpenAI stops new fine-tuning jobs on January 6, 2027. Below is the framework we use on client builds, with current vendor prices, three worked examples and a decision table.

Fine tuning vs RAG vs prompt engineering: the short answer

In the fine tuning vs RAG debate, start with neither: start with the prompt, add RAG when the model lacks facts, and fine-tune only when a measured behavior gap survives good prompting. They fix different failures, and most production systems we build use at least two.

The fastest way to know which one you need is a failure log. Run 50 to 200 real inputs through your current setup, mark each wrong answer, and write one word next to it: knowledge or behavior. Together AI’s RAG vs fine-tuning guide suggests an eval set of about that size too. It turns an architecture argument into a count.

• Mostly knowledge errors (wrong price, outdated policy, invented SKU): build RAG.
• Mostly behavior errors (wrong tone, broken JSON, inconsistent labels): rewrite the prompt, add examples and structured outputs, then re-run the log.
• Behavior errors that survive two rounds of prompt work: now consider fine-tuning.

One row of the log looks like this:

{"id": 41, "input": "Can I return sale items bought in March?",
 "expected": "No. Sale items are final sale (policy v7, section 3).",
 "got": "Yes, within 30 days.",
 "failure": "knowledge", "note": "model never saw policy v7"}

Boring, yes. It also settles the debate in an afternoon.

What each approach actually changes

Prompt engineering changes the instructions, RAG changes what the model can see at answer time, and fine-tuning changes the model’s weights. That difference decides how fast each one updates, what it costs to run, and what can break.

Prompt engineering

Instructions, a few worked examples, a JSON schema for structured output. Nothing to train, and you can change it in minutes. The catch is that every instruction token is paid for on every request, unless you cache it.

RAG (retrieval-augmented generation)

Relevant passages from your documents are fetched and added to the prompt for each question. Update a document and the next answer reflects it, with a citation. The pipeline itself is in our explainer on what RAG is.

Fine-tuning

You train the model on hundreds or thousands of input and output pairs until the behavior is baked in. LoRA and similar methods train only a small slice of the weights, which is why runs got cheap. What it does badly is teach facts: a tuned model can’t cite its source, and when the fact changes you train again.

So prompt engineering vs fine tuning is really about where instructions live. In the prompt you can read and edit them. In the weights you can’t, but you stop paying to resend them.

The 2026 fine-tuning market is smaller than you think

Fewer flagship models can be fine-tuned in late 2026 than a year ago, and that should shape your decision before cost does.

The biggest change is OpenAI. Its deprecations page says organizations that had never fine-tuned lost access on May 7, 2026, and those with no tuned-model inference in the prior 60 days followed on July 2. On January 6, 2027, everyone else loses the ability to create jobs. Existing tuned models keep serving until their base model is retired, and the same page lists GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini for removal on January 14, 2027.

Elsewhere:

• Google still sells supervised tuning for current Gemini models, including Gemini 3.5 Flash, on its Gemini Enterprise Agent Platform pricing page. This is the most practical path to a tuned frontier-class model right now.
• Anthropic models can be fine-tuned only through Amazon Bedrock, and the Bedrock fine-tuning docs list just one Claude model for it: Claude 3 Haiku, in us-west-2.
• Open-weight models (Llama, Qwen, gpt-oss, Gemma) can be tuned on Bedrock, Together AI and similar hosts, or on your own GPUs. Owning the weights matters more now than it used to.

What each one costs in November 2026

Training is now the cheapest line in a fine-tuning budget; inference, hosting and dataset work cost far more. RAG adds retrieved tokens to each request plus a vector store. Prompting costs only tokens. Current list prices, checked this month:

Google’s figures come from its pricing page, which lists tuning per 1,000 tokens ($0.01 for 3.5 Flash) and defines training tokens as dataset tokens multiplied by epochs. Bedrock figures are from the Amazon Bedrock pricing page. The OpenAI row comes from Developers Digest’s fine-tuning tracker, which last checked it against OpenAI’s pricing docs on October 18, 2026. Treat that row as a secondhand figure.

Two lines deserve a second look. The 1.5x premium on tuned Gemini 3 endpoints outside the global region means data residency rules can make your tuned model half as expensive again. And Provisioned Throughput: one model unit at $23.50 an hour, left on all month, is about $17,000. Token-billed tuned inference avoids that trap.

For RAG, five 600-token chunks add 3,000 input tokens per question, or $0.0045 at Gemini 3.5 Flash’s $1.50 per million.

Data needs: what you have to collect first

Prompting needs an eval set. RAG needs clean, current documents. Fine-tuning needs hundreds to thousands of well-labeled examples, and that step sinks most projects.

The labeling row is where we push back hardest. If three support agents would tag the same ticket three ways, the tuned model copies that disagreement at scale. Fix the labeling guide first. Pasted into a prompt, that guide sometimes solves the problem alone.

Latency and maintenance over a year

Fine-tuning is usually fastest per request and RAG the slowest. Over twelve months, though, fine-tuning is the most work to keep alive.

Latency

RAG adds an embedding call, a vector search and often a reranker before generation starts. On our builds that is typically tens to a few hundred milliseconds. Fine-tuning helps with shorter prompts and the option of a smaller, faster model. Caching narrows the gap for static prompts but does nothing for retrieved text, which changes every question.

Maintenance

• Prompts need version control and an eval run on every model upgrade. An hour or two, if the eval set exists.
• RAG needs an ingestion job, freshness checks and an owner per document. Most RAG bugs we fix are content bugs: two versions of one policy in the index.
• Fine-tuned models need the dataset kept current, retraining when behavior requirements change, and a full retrain plus re-evaluation every time the base model is retired. OpenAI’s schedule shows how fast that can come: a fine-tuned o4-mini shut down on October 23, 2026, about 18 months after that base model shipped.

When to fine tune LLM models, and when to skip it

Fine-tune when you need a narrow, repeatable behavior at high volume, have clean labeled examples, and prompting has plateaued on your eval set. Skip it if the problem is facts or volume is low.

Good reasons we’ve seen hold up:

1. Classification and routing at scale. Ticket triage, content moderation, intent detection. Anthropic’s announcement of Claude 3 Haiku fine-tuning reported a moderation test where accuracy rose from 81.5% to 99.6% and tokens per query fell 85%. That’s a vendor number on one task, but it’s the right kind of task.
2. Strict output formats. Extraction into a fixed JSON schema where a missing field breaks something downstream.
3. Downsizing. Teaching a small model a large model’s job, once volume makes the gap real.
4. A voice instructions can’t pin down. Rare, and only with thousands of approved examples.

Bad reasons: “the model doesn’t know our products” (that’s RAG) or “we want it smarter” (it won’t be). When to fine tune an LLM is a volume question as much as a quality one. Below a few thousand requests a day, savings rarely cover the dataset work.

Decision table: RAG vs fine tuning by use case

Find the row closest to your use case and build the “first” column first. This RAG vs fine tuning table is how we scope projects for our AI software services clients.

Worked examples with current prices

All three are hypothetical and use Google’s list prices from the table above. We use Gemini because it’s the frontier family you can still prompt and tune on one platform.

Example 1: ticket triage, where caching beats tuning

Say a retailer routes 100,000 tickets a month into 12 categories. The prompted version on Gemini 3.5 Flash sends a 3,000-token prompt (rules plus 40 examples) and a 400-token ticket, and gets back about 10 tokens. The tuned version needs only a 300-token instruction.

• Prompted, no caching: 340M input tokens at $1.50 per million plus 1M output at $9 comes to $519 a month.
• Prompted, static prefix cached: 300M cached tokens at $0.15, 40M fresh at $1.50, plus output: $114 a month (we’ve ignored cache storage to keep it simple).
• Fine-tuned, global endpoint: 70M input at $1.50 plus output: $114 a month, plus about $24 per training run (2,000 examples of 400 tokens, 3 epochs).
• Fine-tuned, regional endpoint: 1.5x the regional base price: about $188 a month.

On cost, tuning ties at best. It only earns its place here if it’s measurably more accurate on your eval set. That’s the prompt engineering vs fine tuning trade-off in one example: cache first.

Example 2: a store policy assistant, where RAG is the only sane option

Say a Shopify brand gets 20,000 returns, shipping and product questions a month, and its catalog changes weekly. Each question carries about 3,000 retrieved tokens plus 1,000 of instructions, with 300-token answers. That’s 80M input ($120) and 6M output ($54): about $174 a month on Gemini 3.5 Flash, plus the vector store. A tuned model would need retraining every time the returns window changed, and still couldn’t cite the policy. The full build is in our guide to building an AI chatbot for your website.

Example 3: document extraction, where tuning to downsize wins

Say a distributor extracts 1 million supplier documents a month into JSON for its ERP. Prompted on Gemini 3.5 Flash with a cached 2,500-token schema prompt, a 1,500-token document and 400 tokens of output, the bill is about $6,225 a month. A fine-tuned Gemini 2.5 Flash with a 300-token instruction does the same job at $0.30 input and $2.50 output, which is about $1,540 a month.

That’s roughly $4,700 a month saved, against about $150 per training run (5,000 examples of 2,000 tokens, 3 epochs at $5 per million). The real costs are labeling those 5,000 examples and the retrain you’ll owe when Gemini 2.5 Flash retires. Still, this is where fine-tuning wins: high volume, a fixed schema, a cheaper model that can learn the task.

Combining approaches: the hybrid most teams end up with

Most mature systems use all three: a solid prompt as the base, RAG for facts, and a tuned model only for the narrow step that runs at high volume. Add them in that order and measure after each. A typical ecommerce hybrid:

1. A small fine-tuned classifier routes each message (order status, returns, product question, complaint).
2. Order-status questions call the Shopify Admin API for live data instead of trusting the model.
3. Policy and product questions go through RAG over the help center and product metafields.
4. A general model writes the reply under a prompt holding tone and escalation rules.

Each piece fails differently, so measure them separately: retrieval hit rate, classifier accuracy, faithfulness and cost per conversation. Our post on reducing LLM costs covers the routing and caching side.

You can also fine-tune a model to use retrieved passages better, which blurs the fine tuning vs RAG line. We’d try that only after plain RAG plus a good prompt has been measured and found short.

Frequently asked questions

What we’d do first

Build the failure log this week. Fifty to 200 real inputs, the answer you wanted, the answer you got, and one word for why it was wrong. That single sheet tells you more about fine tuning vs RAG for your use case than any comparison article, this one included.

If the errors are about facts, build retrieval and stop there until the numbers say otherwise. If they’re about behavior, spend two rounds on the prompt and turn on caching. Only then price a fine-tune, and price the retrain alongside it. If you’re on OpenAI fine-tuning today, put the January 2027 dates on the roadmap now.

Want a team that has shipped all three to read your log? Our AI software development team can scope it with you.