Retrieval-augmented generation (RAG)

Retrieval-augmented generation, usually called RAG, searches outside the model for useful information before it answers. It can give you a current, specific answer with sources, but your firm has to keep those sources current, control who can read them, and test what the search returns.

You’re giving the model information for this request, rather than training it on your company. If the search finds an old policy, misses the right clause, or returns one client’s file to another client, the answer starts with a problem before the model writes anything.

The paper that named RAG was submitted on May 22, 2020, and current systems can use keyword, semantic, vector, or hybrid search. You can also test how well the evidence supports an answer, with tools such as Google Cloud’s grounding check, which returns a score from 0 to 1.

How does RAG work?

RAG has three parts in its name. The system retrieves information, adds what it found to the current request, and generates an answer from the question and that evidence.

The source material needs preparation before anyone asks a question. AWS documents the common path, convert documents to text, split them into usable chunks, turn those chunks into numbers called embeddings, and store them in an index with links back to the sources.

When a question arrives, the system searches for related chunks and puts the best matches into the model’s input.

Vector search is common because it can match meaning even when the question and source use different words, but it’s only one option. Microsoft’s May 20, 2026 Foundry guidance lists keyword, semantic, vector, and hybrid retrieval.

A price by client ID may be better found with a database query, an exact policy phrase with keyword search, and a broad staff question with semantic search, which looks for meaning. Choose the method and source that can answer the question.

Consider an internal procedure assistant. A staff member asks whether a client deliverable needs a second review. The system retrieves the current review policy, adds the relevant section to the request, and returns a short answer with a link to the policy. If no approved source answers the question, the right result is a refusal or a handoff.

What changes when RAG uses company knowledge?

The source set becomes part of delivery. A model vendor may supply the generator. A software vendor may supply the index. Neither vendor owns the policy that changed on Tuesday, the duplicate file from last quarter, or the access rule that separates two clients.

When you ask the model alone, it has its training and the current prompt, so company facts may be missing or old. If you attach documents by hand, you’re still the person who has to find the right files and carry them into the request.

With RAG inside a workflow, the workflow searches approved sources when the question arrives. Someone still has to maintain the search, access rules, and source quality, but the lookup no longer depends on a person gathering files for every request.

That’s why I wouldn’t start by uploading a whole folder. A shared drive usually has drafts, final versions, duplicates, scans, and files nobody has owned for two years, and indexing all of them makes them easier to find without deciding which version to trust.

The first useful source set is narrow. Start with one approved policy library or one current set of operating procedures. Signed scopes and client decisions can work when access follows the account. Each source needs an owner who can name the current version, say when it changed, and define what happens when the system finds no answer.

Does RAG stop hallucinations?

RAG can reduce unsupported answers by giving the model relevant evidence. It does not ensure that the evidence is complete or that the model uses it correctly.

Microsoft’s Foundry documentation names several limits. Poor content preparation and retrieval settings reduce answer quality. Irrelevant or incomplete passages can lead to an incomplete or inaccurate answer. Grounding can also use too much of the model’s token budget, add delay, and increase cost.

A citation helps because a reviewer can open the source, but it doesn’t prove the sentence matches it. The passage may support only half the claim, come from an old version, or disagree with another source that the model has quietly combined into one smooth answer.

Google Cloud’s grounding check tests one part of this problem, returning a support score from 0 to 1 and citations for the claims it finds. A partly supported sentence doesn’t count as grounded in that check, which is useful, but still only one part of how your firm checks the work.

The system needs an allowed answer for weak evidence: “I could not find an approved source for this.” An assistant that must answer every question will turn a retrieval miss into a guess.

How is RAG different from fine-tuning, memory, and a context window?

These names describe different parts of an AI system.

Fine-tuning changes model behavior through additional training. It can make a model follow a format or perform a repeated task more consistently. It is a poor way to carry a policy that changes each month. RAG can retrieve the current policy at answer time without retraining the model.

AI memory is information saved outside the current request for later use. Retrieval selects stored information for the job in front of the system. A client preference in memory helps only when a later workflow finds it, checks its scope, and brings it into the current request.

The context window is the amount of information the model can consider in one request. Every retrieved passage uses part of it. Sending twenty weak passages can crowd out the question, instructions, or one section that matters. More retrieval can lower answer quality.

RAG can supply evidence inside an AI workflow or an agent. The workflow still decides when to search, which sources are allowed, how to check the result, and who handles an answer that fails.

What should a business put into a RAG system?

Start with records that answer one frequent, bounded question. Staff policy questions, approved operating procedures, current service boundaries, or a client’s signed scope can fit. A request to “understand everything in our Drive” cannot.

For each source, name someone who can correct it, mark the approved version, decide which date or event triggers an update, set which users and clients can retrieve it, and keep a source location the reviewer can open.

Keep the document’s structure when you prepare it, so a heading stays with its paragraph and a table doesn’t leave the price in one chunk and the service name in another. A scanned PDF also needs usable text before the system can search it, and shortcuts here can show up in what gets retrieved.

Old information needs a removal path. Deleting a file from the source system is incomplete if its chunks remain searchable in the index. A policy update must replace or retire the old evidence. Then run the known questions again.

How do permissions work in RAG?

Permissions apply before the system returns a passage. Filtering after generation is too late because the model has already seen the protected information.

Microsoft advises applying access control at retrieval time. Its guidance also says to treat retrieved content as untrusted input. A document can contain instructions meant to steer the model, including text placed there by a client, employee, or outside party.

For a business, client scope is the first hard boundary. Someone allowed to work on Client A should not retrieve Client B’s proposal, price, or meeting note because both files mention the same service. Semantic similarity does not grant permission.

Use the user’s identity and the source system’s access rules when searching, and log which passages came back. Test a question the user isn’t allowed to answer on purpose.

When someone loses access or a client leaves, check that retrieval stops returning those records. Even a correct, well-supported answer is a security failure if it came from a document the user wasn’t allowed to read.

How should a firm test RAG?

Test retrieval and generation separately, because finding the wrong passage and misreading the right one need different fixes.

Build a small question set from real work. For each question, record the approved source and the expected answer or refusal. Include an easy lookup, two documents that disagree, an outdated version, a missing answer, a cross-client question, and a document containing an instruction the system should ignore.

For retrieval, check whether the right evidence appeared near the top and forbidden evidence stayed out. For the answer, check whether each claim is supported, the citation opens the right source, and the system refused when it couldn’t support an answer.

An eval makes those checks repeatable. The same evidence helps you decide how far to delegate the inputs while still owning the result.

The original 2020 RAG paper showed why external memory was promising. Its models combined a generator with a dense Wikipedia index and set the state of the art on three open-domain question-answering tasks. That result belongs to those tasks and that system. A business still needs tests built from its own policies, clients, language, and failure costs.

Where does RAG sit on the Delivery Model Ladder?

RAG sits at Stage 2, Augmented, on the Delivery Model Ladder when a delivery workflow retrieves current company records under the user’s permissions, checks the answer, preserves its sources, and routes weak evidence to a named person.

At Stage 1, Enhanced, a person chooses the files, asks the question, reads the citations, and carries the answer into the next system. The model may use retrieval. The person still holds the workflow together.

The Production Gap stays open until someone owns the sources, index updates, access filters, retrieval tests, and failed answers. RAG works best for specific questions over records your firm can keep current.

Use ordinary search or a database query when either gives you the answer more directly. And use a qualified person when the work needs judgment across a whole contract, financial record, or technical file.