If a policy changes, a document assistant needs the new policy, not just a different writing style. If the assistant repeatedly formats results incorrectly, retrieval alone may not fix that. Separate the information problem from the behavior problem before choosing an approach.
The terms are often presented as competing model strategies, but they change different parts of the system. RAG adds an external retrieval path at inference time. Fine-tuning changes model parameters through training examples. The original RAG research described a combination of parametric and non-parametric memory; modern enterprise implementations also need ingestion, permissions, citations, evaluation, monitoring, and a process for updating or removing knowledge.
Related service: AI & Automation services- 01Define whether the gap is knowledge, behavior, latency, cost, or several of these.
- 02Create a representative evaluation set before changing the architecture.
- 03Establish a prompt-and-context baseline on an appropriate base model.
- 04Test retrieval when answers depend on changing or private sources.
- 05Test fine-tuning when stable examples define the desired behavior.
- 06Combine approaches only if each produces a measured improvement worth its build and running costs.
- 07Monitor source freshness, retrieval quality, answer quality, safety, latency, and unit cost after launch.
SECTION 01
Start with the problem, not the technique
A support assistant that cannot see the current policy has a knowledge problem. A classifier that repeatedly produces the wrong output structure may have a behavior problem. A system can also fail because its task definition, base model, prompt, tools, or evaluation is weak. Diagnose that layer before paying for more architecture.
| Decision factor | RAG | Fine-tuning | RAG + fine-tuning |
|---|---|---|---|
| Primary need | Current, private, or attributable knowledge | Consistent task behavior from stable examples | Specialized behavior grounded in changing knowledge |
| Knowledge update | Re-index approved content without retraining the model | Curate a new dataset and repeat training and evaluation | Update retrieval often; retrain only when behavior needs to change |
| Source visibility | Can return passages and document references | Training examples are not a reliable per-answer citation mechanism | Use retrieval evidence for answer attribution |
| Main data asset | Clean, permissioned, versioned source corpus | Representative, reviewed input-output examples | Both assets with separate owners |
| Typical failure | Wrong or missing retrieval, stale content, context overload | Weak examples, overfitting, regression, learned unwanted behavior | More interacting layers and harder diagnosis |
| Operational burden | Ingestion, chunking, indexing, access filters, freshness, retrieval evals | Dataset lineage, training runs, model versions, regression evals | Both, plus orchestration and attribution |
SECTION 02
Choose RAG for current, private, and attributable knowledge
A RAG pipeline prepares approved content, creates searchable representations, retrieves relevant passages for a request, and supplies that evidence to the generation step. It is a strong fit for policies, product documentation, operating procedures, support knowledge, contracts, service records, or other sources that change independently of the model.
Retrieval does not guarantee a grounded answer. The system may retrieve nothing, retrieve the wrong passage, cross a permission boundary, or provide conflicting versions. Evaluate retrieval recall and relevance separately from whether the final response used the evidence correctly. Preserve document identity, version, effective date, access rules, and deletion behavior through the ingestion pipeline.
SECTION 03
Choose fine-tuning for stable, demonstrable behavior
Fine-tuning is useful when many reviewed examples show a consistent transformation the base model does not perform reliably enough. Examples include classification labels, a strict response structure, specialized tone, recurring extraction behavior, or following a domain-specific instruction pattern. OpenAI's model-optimization guidance frames fine-tuning as one part of an evaluation-driven optimization cycle, not a replacement for evals.
The dataset must represent real variation, edge cases, refusals, and desired boundaries—not just ideal examples. Separate training and test data, protect sensitive information, preserve lineage and reviewer decisions, and compare against the baseline. If the underlying fact changes frequently, place it in retrievable data rather than expecting retraining to act as a content-management system.
Good signal
Reviewers can consistently demonstrate the desired output across enough representative cases.
Weak signal
Stakeholders disagree about the correct answer or change the task after seeing each result.
Good boundary
The behavior is stable while changing facts remain in prompts, tools, or retrieval.
Weak boundary
The proposal expects model weights to store a frequently updated catalog, policy set, or customer record.
SECTION 04
Combine RAG and fine-tuning only for two proven gaps
A combined design can make sense when a system must follow a specialized behavior while answering from a changing corpus. For example, retrieval may supply current technical procedures while a fine-tuned model produces a consistent structured diagnostic record. The retrieved evidence still owns the facts; the fine-tuned behavior owns the transformation.
Avoid combining techniques before establishing a baseline. Add retrieval, rerun the same evaluation set, and classify remaining failures. If a repeatable behavior gap remains and high-quality examples exist, test fine-tuning. This sequence shows whether each layer earns its complexity and prevents a retrieval failure from being misdiagnosed as a model-training problem.
- 01
Measure the prompt-and-base-model baseline.
- 02
Add permission-aware retrieval and measure retrieval plus answer quality.
- 03
Inspect remaining failures for a stable, example-driven behavior gap.
- 04
Fine-tune on reviewed examples and rerun regression, safety, and cost tests.
- 05
Keep separate version records for corpus, index, prompt, model, tools, and evaluation set.
SECTION 05
Compare lifecycle work, not one API price
RAG cost includes source inventory, parsing, cleaning, metadata, access filters, indexing, retrieval calls, generation, storage, reprocessing, monitoring, and content ownership. Fine-tuning cost includes example collection, expert review, privacy checks, dataset versioning, training, model hosting or usage, regression evaluation, and retraining decisions. A combined system carries both sets of work.
Build a cost model around the actual workload: documents and change rate for RAG; examples and behavior change rate for fine-tuning; request volume, context size, latency target, model choice, and review rate for either. Record assumptions and run low, expected, and high scenarios. A technically cheaper response is not a saving if retrieval misses the right source or reviewers must repair the output.
| Asset | Owner question | Evidence to track |
|---|---|---|
| Source corpus | Who approves, updates, restricts, and retires content? | Version, effective date, access policy, ingestion status |
| Training examples | Who judges correctness and resolves disagreement? | Lineage, reviewer, split, inclusion rationale |
| Model and prompt | Who approves upgrades and rollback? | Version, change record, evaluation comparison |
| Evaluation set | Who adds real failures without contaminating tests? | Case source, expected result, rubric, severity |
| Production workflow | Who handles no-answer, conflict, unsafe, and integration failures? | Queue ownership, response target, outcome, recurrence |
SECTION 06
Evaluate quality, permissions, and failure behavior
For RAG, test whether the correct source is available, permitted, retrieved, and cited, then test whether the answer is supported by it. For fine-tuning, test task behavior on held-out cases, difficult variations, safety cases, and known regressions. For both, test missing context, conflicting instructions, malicious retrieved content, prompt injection, sensitive-data requests, and tool failures.
Use the same business test set to compare approaches. Report results by category and severity, not only one aggregate score. Monitor production drift in source coverage, retrieval results, task mix, reviewer corrections, latency, and cost. Re-evaluate when models, prompts, indexes, tools, permissions, source content, or policies change.
SECTION 07
Frequently asked questions
Is RAG better than fine-tuning for enterprise data?
RAG is generally the stronger fit when answers depend on current, private, removable, or citable enterprise sources. Fine-tuning is stronger when reviewed examples define a stable behavior. The correct choice depends on an evaluation of the actual task.
Can fine-tuning replace a company knowledge base?
Fine-tuning should not be treated as a content-management system for frequently changing facts. Put current policies, catalogs, records, and documents in governed prompts, tools, or retrieval sources; use fine-tuning for stable behavior.
Can a company use RAG and fine-tuning together?
Yes, when tests show both a knowledge gap and a stable behavior gap. Establish a baseline, add and evaluate retrieval first, then fine-tune only if representative examples address remaining failures.
What should be evaluated in a RAG system?
Evaluate source availability, permissions, retrieval recall and relevance, no-answer behavior, conflict handling, citation correctness, answer groundedness, safety, latency, reviewer corrections, and cost. Retrieval and generation should have separate scores.
PRIMARY REFERENCES
Sources and further reading
These references cover the standards, platforms, or published prices discussed in the guide. Worked examples and checklists are our editorial guidance.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al.
- Retrieval guideOpenAI
- Model optimization and fine-tuningOpenAI
- Working with evalsOpenAI
- Generative AI Profile (NIST AI 600-1)National Institute of Standards and Technology
- OWASP Top 10 for LLM and GenAIOWASP GenAI Security Project
EDITORIAL METHOD
About this guide
We use AI to assist with drafting and editing. Catapult AI Work is responsible for the published content. Examples illustrate possible approaches; they are not client case studies unless identified as such.
Budget examples are not Catapult package prices. Check linked provider pages for current fees and plan limits before making a purchase.
Read the editorial policy