How documents are split into chunks and indexed

The assistant doesn't search your file as one block. It searches small pieces of it. If the right piece wasn't picked, the answer won't use it, even when the next paragraph over was a perfect match.

Why files get cut up

Language models can only take in so much text at once. Microsoft's documentation says that partitioning large documents into smaller chunks can help you stay under the maximum token input limits of the models.¹ A token is the small unit of text a model counts.

There's a second reason. Microsoft's search documentation says to use chunking to subdivide large documents so that portions can be matched on independently.² A long handbook covers many topics. A question about parental leave shouldn't be matched against the whole handbook.

So the tool slices each file into chunks. Then it builds an index, which works like the index at the back of a book. It's a lookup structure that lets the system find matching chunks quickly instead of rereading everything.

Where the cuts fall

Different tools cut in different ways. Microsoft lists a few common approaches.¹

Anthropic notes that the choice of chunk size, chunk boundary and chunk overlap can affect retrieval performance.³ So two tools given the same file can find different things.

Why the paragraph next door gets missed

Picture a price list cut so that a heading lands in one chunk and the prices land in the next. The search sees a chunk full of numbers with no label. Your question about "wedding cakes" matches the heading chunk, not the prices.

Anthropic describes a similar case. A chunk may not say which company it's talking about or what time period it covers, and that makes it hard to retrieve the right information or use it well.³ It also says traditional approaches often remove context when encoding information.³

Makers have a few fixes. Microsoft describes overlap, where a small amount of text is repeated between neighboring chunks to help maintain continuity and context.¹ It also describes adding the document title to chunks from the middle of a document to prevent context loss.¹ You may not be able to change these settings in a simple product, but it helps to know they exist.

What you can do as a document owner

You usually can't see the chunks. You can still write files that survive being cut.

• Put the subject in the same paragraph as the detail. "Weekend delivery costs extra" works better than a table whose heading is three pages earlier.
• Use clear headings, since some tools cut along them.
• Keep one topic per section.
• Repeat the product or policy name where the detail sits, instead of writing "it" or "the above."

Microsoft says RAG quality depends on how you prepare content for retrieval.² Preparation includes how you write the file.

If the assistant keeps missing something you know is in a file, test it. Ask the question using the exact wording from the file. If that works but your natural wording doesn't, the problem is likely matching, which embeddings and meaning-based search explains. If neither works, the passage may not have been picked at all. Why answers go wrong lists more causes.