What retrieval means when an AI assistant answers from your files

When a chatbot says it answers from your files, it does two jobs in order. First it looks up passages. Then it writes a reply using what it found. The looking-up part is called retrieval.

The short version

The technical name for the whole approach is retrieval-augmented generation, usually shortened to RAG. Amazon describes it as a process that makes a language model reference "an authoritative knowledge base outside of its training data sources before generating a response".¹

A language model (often called an LLM, short for large language model) is the part that writes the sentences. On its own it only knows what it picked up during training. Retrieval hands it extra reading material at the moment you ask.

Wikipedia puts it this way: the model will first refer to a specified set of documents, then respond to the question.² That order matters. The lookup comes before the writing.

What happens when you ask a question

Think of a new staff member who gets a question at the front desk. They don't answer from memory alone. They walk to the filing cabinet, pull a few pages, and read them before replying. Retrieval is the walk to the cabinet.

Here's the usual order, in plain terms:

• You type a question.
• The system searches the material that was added to it, such as your price list or staff handbook.
• It picks out the passages that look most relevant.
• It sends your question and those passages to the language model together. AWS says "The user query and the relevant information are both given to the LLM."¹
• The model writes a reply, using the passages plus what it already knew from training.

The pasted-in passages are the whole trick. Anthropic describes RAG as a method that retrieves relevant information from a knowledge base and appends it to the user's prompt.³ The prompt is just the text the model receives.

Why bother with it

Without retrieval, a model answers from training alone. AWS lists a known problem. Models can present "out-of-date or generic information when the user expects a specific, current response."¹

Your own opening hours, return rules and service list were never in its training. Retrieval gives the model a way to see them. Wikipedia notes that this enables LLM-based chatbots to access internal company data.²

What retrieval doesn't do

It doesn't make the assistant read your whole library every time. It fetches a few passages, and the answer can only be as good as those passages. If the search picks the wrong ones, the reply may sound sure of itself and still be off.

It also doesn't change the model. Your files are read at question time, not learned. The page on a knowledge base versus training explains that difference.

If you see an answer that seems to ignore a document you uploaded, one likely cause is that the lookup step missed the right passage. The page on splitting files into chunks shows why that happens. Other causes are in the page on wrong answers.

Words you'll see

• Knowledge base: the collection of your documents the assistant is allowed to search.
• Query: your question, as the search system receives it.
• Retrieve: to find and fetch matching passages.
• Prompt: the full text sent to the model, including your question and any fetched passages.

Tools use different labels for these. If a product says "grounded in your data," "answers from your files" or "sources," it's usually describing some form of this lookup. Check the maker's own documentation for exactly what your tool does.