Retrieval-augmented generation (RAG)

Retrieval-augmented generation, usually called RAG, searches outside the model for useful information before it answers. The paper that named RAG was submitted on May 22, 2020. Current systems can use keyword, semantic, vector, or hybrid search. Google Cloud’s grounding check returns a score from 0 to 1 for support from supplied facts. Those details point to the operator’s real job. RAG does not teach the model your company. It creates a search path over records the firm must keep clean, current, permissioned, and testable. A good path returns a current, specific answer with sources. A bad path retrieves an old policy, misses the right clause, or exposes one client’s file to another for that request.

How does RAG work?

RAG has three parts in its name. The system retrieves information, adds what it found to the current request, and generates an answer from the question and that evidence.

Prepare the source material before anyone asks a question. AWS documents the common path. Convert documents to text. Split them into usable chunks. Turn those chunks into numbers called embeddings. Store them in an index while keeping the link back to the source. When a question arrives, search for related chunks and place the best matches in the model’s input.

Vector search is common because it can match meaning when the question and source use different words. It is one option. Microsoft’s May 20, 2026 Foundry guidance lists keyword, semantic, vector, and hybrid retrieval. A price by client ID may belong in a database query. An exact policy phrase may need keyword search. A broad staff question may benefit from semantic search. The useful system sends each question to the source that can answer it.

Consider an internal procedure assistant. A staff member asks whether a client deliverable needs a second review. The system retrieves the current review policy, adds the relevant section to the request, and returns a short answer with a link to the policy. If no approved source answers the question, the right result is a refusal or a handoff.

What changes when RAG uses company knowledge?

The source set becomes part of delivery. A model vendor may supply the generator. A software vendor may supply the index. Neither vendor owns the policy that changed on Tuesday, the duplicate file from last quarter, or the access rule that separates two clients.

Setup	Where the information comes from	Who starts the lookup	Main limit
Model-only answer	The model’s training and current prompt	A person	Company facts may be missing or old
Manual document use	Files a person selects for this request	A person	The person still finds and carries the context
RAG in a workflow	Approved sources searched at answer time	The workflow	Retrieval, access, and source quality must be operated

Uploading a folder is a weak first implementation. A shared drive usually holds drafts, final versions, duplicates, scans, and files nobody has owned for two years. Indexing all of it makes each version easier to retrieve. It does not decide which version is true.

The first useful source set is narrow. Start with one approved policy library or one current set of operating procedures. Signed scopes and client decisions can work when access follows the account. Each source needs an owner who can name the current version, say when it changed, and define what happens when the system finds no answer.

Does RAG stop hallucinations?

RAG can reduce unsupported answers by giving the model relevant evidence. It does not ensure that the evidence is complete or that the model uses it correctly.

Microsoft’s Foundry documentation names several limits. Poor content preparation and retrieval settings reduce answer quality. Irrelevant or incomplete passages can lead to an incomplete or inaccurate answer. Grounding can also use too much of the model’s token budget, add delay, and increase cost.

A citation helps because a reviewer can inspect the source. It does not prove that the sentence matches the source. The passage may support half the claim. It may come from an old revision. The model may combine two sources that disagree and hide the conflict in smooth prose.

Google Cloud’s grounding check puts one part of this problem into a test. It returns a support score from 0 to 1 and citations for the claims it finds. A partly supported sentence does not count as grounded in that check. That is useful evidence. It remains one part of the firm’s validation method.

The system needs an allowed answer for weak evidence: “I could not find an approved source for this.” An assistant that must answer every question will turn a retrieval miss into a guess.

How is RAG different from fine-tuning, memory, and a context window?

These names describe different parts of an AI system.

Fine-tuning changes model behavior through additional training. It can make a model follow a format or perform a repeated task more consistently. It is a poor way to carry a policy that changes each month. RAG can retrieve the current policy at answer time without retraining the model.

AI memory is information saved outside the current request for later use. Retrieval selects stored information for the job in front of the system. A client preference in memory helps only when a later workflow finds it, checks its scope, and brings it into the current request.

The context window is the amount of information the model can consider in one request. Every retrieved passage uses part of it. Sending twenty weak passages can crowd out the question, instructions, or one section that matters. More retrieval can lower answer quality.

RAG can sit inside an AI workflow or an agent. It supplies evidence. The workflow decides when retrieval runs, which source it may search, what check follows, and where a failed answer goes.

What should a service firm put into a RAG system?

Start with records that answer one frequent, bounded question. Staff policy questions, approved operating procedures, current service boundaries, or a client’s signed scope can fit. A request to “understand everything in our Drive” cannot.

Each source needs five details:

• an owner who can correct it;
• a status that marks the approved version;
• a date or event that triggers refresh;
• a scope that says which users and clients may retrieve it;
• a source location a reviewer can open.

Document structure matters. A heading should travel with the paragraph under it. A table should not be split so the price lands in one chunk and the service name in another. A scanned PDF needs usable text before the system can search it. Retrieval results will show every shortcut in this preparation.

Old information needs a removal path. Deleting a file from the source system is incomplete if its chunks remain searchable in the index. A policy update must replace or retire the old evidence. Then run the known questions again.

How do permissions work in RAG?

Permissions apply before the system returns a passage. Filtering after generation is too late because the model has already seen the protected information.

Microsoft advises applying access control at retrieval time. Its guidance also says to treat retrieved content as untrusted input. A document can contain instructions meant to steer the model, including text placed there by a client, employee, or outside party.

For a service firm, client scope is the first hard boundary. Someone allowed to work on Client A should not retrieve Client B’s proposal, price, or meeting note because both files mention the same service. Semantic similarity does not grant permission.

Use the user’s identity and the source system’s access rules in the retrieval query. Log which passages were returned. Test a forbidden question on purpose. When a person loses access or a client leaves, verify that retrieval stops. A well-supported answer from a forbidden document is a security failure.

How should a firm test RAG?

Test retrieval and generation separately. A bad answer may start because the system found the wrong passage. It may also start because the model misread the right one. Those failures need different fixes.

Build a small question set from real work. For each question, record the approved source and the expected answer or refusal. Include an easy lookup, two documents that disagree, an outdated version, a missing answer, a cross-client question, and a document containing an instruction the system should ignore.

The retrieval test asks whether the right evidence appeared near the top and whether forbidden evidence stayed out. The answer test asks whether each claim is supported, whether the citation opens the right source, and whether the system refused when support was weak. An eval turns those checks into a repeatable score. The same evidence tells the firm how far it can delegate the inputs while keeping ownership of the result.

The original 2020 RAG paper showed why external memory was promising. Its models combined a generator with a dense Wikipedia index and set the state of the art on three open-domain question-answering tasks. That result belongs to those tasks and that system. A service firm still needs tests built from its own policies, clients, language, and failure costs.

Where does RAG sit on the Delivery Model Ladder?

RAG sits at Stage 2, Augmented, on the Delivery Model Ladder when a delivery workflow retrieves current company records under the user’s permissions, checks the answer, preserves its sources, and routes weak evidence to a named person.

At Stage 1, Enhanced, a person chooses the files, asks the question, reads the citations, and carries the answer into the next system. The model may use retrieval. The person still holds the workflow together.

The Production Gap stays open until someone owns the source set, index refresh, permission filter, retrieval test, and failed answer. That ownership is the production boundary. RAG works best for bounded questions over records the firm can keep current. Use ordinary search or a database query when either can return the answer more directly. Use a qualified person when the work requires judgment across a whole contract, financial record, or technical file.