RAG vs fine-tuning: which approach fits your business?

Separate knowledge access from model behaviour

Retrieval-augmented generation, or RAG, fetches relevant material and supplies it to a model as context. Fine-tuning changes model parameters using training examples. They address different parts of an application and can be combined.

AWS highlights RAG as a starting point for questions over custom documents and describes the option of combining it with fine-tuning. Its comparison is a useful architectural reference, not a substitute for evaluating your own task. See AWS’s comparison of RAG and fine-tuning.

When retrieval is the better first experiment

Consider an internal assistant that answers questions about product manuals revised every month. The immediate requirement is to retrieve the correct edition and show the passage behind an answer. Putting those manuals into a training exercise does not remove the need to manage document versions or user permissions.

Build a representative set of questions with known supporting passages. First check whether the application retrieves the right material. Then check whether the generated answer accurately uses it. Treat retrieval failure and answer-generation failure as separate problems.

When to investigate fine-tuning

Fine-tuning is worth investigating when the required behaviour remains inconsistent after a strong baseline using prompts and examples. An example might be a specialised classification task with well-labelled training data and a stable output definition.

Before investing, ask whether reviewers agree on the desired answer, whether training examples represent real inputs and whether evaluation examples can be kept separate. If the labels are inconsistent, a more elaborate training process may reproduce that inconsistency.

Compare complete operating costs

• For retrieval: include document ingestion, indexing, retrieval, generation, permissions and source maintenance.
• For fine-tuning: include dataset preparation, evaluation, training, serving and future retraining.
• For both: include human review, monitoring, failed requests and the process for releasing model changes.

Make the choice with an error log

Run the alternatives against the same held-out tasks. Label each failure: missing source, wrong source, ambiguous instruction, inconsistent formatting or unsupported answer. The distribution of errors points towards the intervention most likely to help.

Neither approach guarantees correct answers or access control. Source citations need to support the associated claim, and a cited document may still be obsolete. Make refusal behaviour and restricted-content tests part of the evaluation, rather than assuming that adding retrieval solves them.