Original URL: https://getagenttools.com/blog/rag-vs-fine-tuning/ # RAG vs Fine Tuning: Choose by Facts, Behavior and Evaluation 11 Oct 2026Marek. - rag - fine-tuning - llm-evaluation An assistant quotes last month's price. Before any talk about RAG vs fine tuning I want one answer. Did it ever see this month's price? Second case. It saw the correct price and returned invalid JSON. Why did the format fail? Put both under "the model needs improving" and it is easy to build something that fixes neither. The architecture decision starts with the wrong answer in front of you. Two questions. What information did the model have? What did it do with it? Retrieval supplies evidence at answer time. Fine-tuning changes learned behavior through examples. Which of those helps depends on what was missing. - **The failed answer.** Missing evidence needs one repair. Mishandled evidence needs another. - **Three starting points.** Changing facts, inconsistent behavior, or a job that needs both. - **The source owner.** Someone has to publish the current record and withdraw the old one. - **A small fixture.** Updating a dictionary value tests freshness logic, not model quality. - **The decision check.** The added component has to improve the same held-out cases. [What is RAG](https://getagenttools.com/blog/what-is-rag/) covers the retrieval mechanism. Here I want to stay with the decision: which change is justified by the failure you can actually show? ## Keep the wrong answer and its context together A support assistant quotes an old fee because the current policy never reached its context. Another receives the correct fee but returns the wrong response structure. A third gets the right policy and misreads it. In a dashboard the three may look similar. The repair is not the same for all three. Get a few actual failures on the table, each with its question, its expected result and whatever the model was given. That last part matters. Leave it out and a wrong answer can fool you. Retrieval looks broken, yet the right passage was already there. Or the model looks incapable, yet nobody gave it the fact it needed. I have a case of my own. On 23 September 2026 my agent asked a fetch tool to quote a site's terms exactly and name the operator. Back came a neat line with a company name. That name went into a data file and on to the designer. A reviewer then found a different operator in the saved page. The company from the neat line had another role there. The tool answers through a small model, and that model had summarized. OpenAI's [accuracy optimization guide](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) separates context problems from learned behavior and distinguishes retrieval failures from failures to use correct context. I use that distinction before any feature list. Then I know where to look next. Before any new component, write one clear prompt. Give it the necessary information. State the output requirement. Run it. That result is your baseline. If it solves the case, you learned something about the failure. And there is no index or training dataset to maintain. ## Who will keep the fix working? | Scenario | First approach to evaluate | Owner and maintenance question | Acceptance check | | --- | --- | --- | --- | | Facts change frequently | Retrieval with controlled source updates | Who publishes and withdraws authoritative records? | Does the answer use the current authorized source? | | Facts are supplied, output behavior is inconsistent | Prompt/schema controls first; fine-tuning if the remaining failure justifies it | Who curates examples and reviews model changes? | Does unseen input produce the required behavior? | | Fresh facts and consistent behavior are both required | Retrieval plus behavior controls; consider fine-tuning separately | Who owns each layer and their shared evaluation? | Is the answer current, grounded and correctly formed? | The maintenance column is part of the decision. A retrieval pipeline with nobody responsible for withdrawing an old policy will eventually supply an old policy. A training dataset with nobody checking the labels will carry those labels into the next model version. Something close happened in my own notes. A number was written down from memory, not copied from the source. A panel showed Approved (6). The number that reached the README and the agent's memory was 3. That wrong state was saved for the next session. A more elaborate architecture fixes neither the stale policy nor the bad labels. An application full of documents can still have a question-routing problem. A classifier can still have unclear labels. Use the table to start investigating. "We have lots of documents" should not settle the implementation. Ask the source owner about authority, versions and access. They should be able to explain all three. The training owner gets two questions. Where did the examples come from? What accepts the next model version? If there is no answer, that work has not disappeared. It still waits for someone after launch. ## When should I choose RAG instead of fine-tuning? I would evaluate RAG first in three cases. The answer needs changing facts. It needs citations. It needs a user's authorized documents. Keep that material outside the model and supply the relevant evidence for the question. Then check that the retrieved evidence is current and permitted. Choosing retrieval does not establish either property on its own. A weekly pricing change is a useful example. The current record can reach the answer without another training operation. But an indexed copy needs updating too, and obsolete material needs a withdrawal path. Otherwise the system retains the same stale answer through a different mechanism. OpenAI's [retrieval documentation](https://developers.openai.com/api/docs/guides/retrieval) describes search and attribute filtering. Restriction by file attributes is in there too. That helps once you have settled what the attributes mean. The application still decides two things. Which version is authoritative. Which user can receive it. And does the question need semantic search at all? With an exact product ID in a small catalog, a direct lookup may do. A broader natural-language question may need another retrieval method. Put the question shapes you expect into the evaluation set before buying infrastructure for the question shapes you imagine. ## A price change without a model call The local fixture prepared for this article contained two synthetic records: `price = 12 credits` and `format = JSON`. A deterministic key lookup returned an active record's value or Unknown. The three probes asked for price, format and a missing refund policy. | State | Price probe | Format probe | Refund probe | | --- | --- | --- | --- | | Initial records | 12 credits | JSON | Unknown | | Price updated to v2 | 15 credits | JSON | Unknown | | Price record withdrawn | Unknown | JSON | Unknown | The local assertions passed. Updating the price changed the returned value without touching the format record. Marking the price inactive made it unavailable. The missing refund policy stayed Unknown. That is the whole result. It checks update and withdrawal logic in a dictionary. No embeddings ran. No semantic ranking. No generation, no training. No model had to decide whether to abstain. It proves nothing about accuracy, speed or cost against fine-tuning. I like a check this small before adding generation. It isolates one question. The answer should not be ambiguous. Did the right source value reach the next step? Add competing versions and unauthorized records to that source-layer test. You can already identify an error here. A fluent answer from a model later should not conceal it. ## Does fine-tuning keep facts up to date? No. Fine-tuning gives a model no live update connection. Its behavior comes from the training process. A new source fact still needs an update path, such as retrieval or supplied context. Train on factual examples and you have to test freshness. A changed document will not hand it to you. OpenAI's [supervised fine-tuning documentation](https://developers.openai.com/api/docs/guides/supervised-fine-tuning) is about learning from input/output examples. Formats are in it. So is instruction-following behavior. That can address a stable behavior requirement. I would not treat it as the update mechanism for a catalog that changes every week. There is a platform constraint to check before making a plan around that particular service. Checked on 3 October 2026. OpenAI says its self-serve fine-tuning platform is winding down. New users cannot get it. The [deprecation timeline](https://developers.openai.com/api/docs/deprecations#update-to-openais-self-serve-fine-tuning) lists restrictions for existing customers. For active customers, new jobs end on 6 January 2027. The notice is about that named platform. The method has not disappeared. Before you plan, look up four things for what you actually intend to use. Base model. Provider. Account eligibility. Lifecycle. If your account cannot use the proposed training path, a conceptual comparison is of little help. I ran into the account part myself, with inference rather than training. On 22 September 2026 the inference profile for a newer model was active in my cloud region. The call still came back with an AccessDenied error. It said the model is not available for this account. The quota for that model was 0. ## Can I use RAG and fine-tuning together? Yes, where the chosen platform supports it. Retrieval can supply current evidence. If a behavior problem remains, fine-tuning can address it. The user sees one answer. In the test record, keep the two requirements separate. Take a policy classifier. Which policy applies is decided in the source layer. The behavior requirement determines how the classification and explanation should be expressed. Test the base model with good instructions on that same evidence first. A persistent behavior gap may remain. That gap is your reason to evaluate training. The earlier run is your baseline. Change the retriever and the model at once and it gets harder to tell which one improved the case. Test one layer. Test the other. Then test the full path. Bad retrieval should not survive behind a good final score just because the model happened to guess the answer. Also include a conflict: a learned example suggests an old policy, while retrieved evidence supplies the current policy. Which should govern? Write that rule into the requirement and check the answer. Combine components and you have one more interaction to inspect. Combining them does not establish how that interaction will resolve itself. ## Which evaluation should I run before deciding? Use a fixed, held-out set with current facts, missing evidence, source changes and the required output behavior. Keep training examples out. Grade retrieval separately from the final answer, and report correctness, grounding, format, abstention and operating cost separately. One aggregate score can hide the very failure you started with. Each case should say which source ID or version was expected. Inspect what actually reached the model before grading what it produced. Wrong evidence points you toward the source or retrieval layer. Correct evidence with a wrong answer gives you a different problem to investigate. The [LLM evaluation workflow](https://getagenttools.com/blog/llm-evaluation/) covers building that set and reviewing the results. Deterministic checks fit things such as required keys and valid output format. Ambiguous factual or reasoning judgments need human review; a model judge needs calibration against cases a person assessed. Finally, put maintenance beside inference cost. Retrieval needs source updates and index checks. Training needs curated data, platform access and acceptance of each model version. A hybrid needs both kinds of work. I would choose the smallest supported change that fixes the observed failures, then keep those cases. They are how you notice the fix stopped working after the next source update or model change. Put this into practice [See the tool list](https://getagenttools.com/tools/)[Train your team](https://getagenttools.com/#training)