Keeping a knowledge base current when documents change

Editing a file on your computer doesn't automatically change what the assistant says. The assistant reads from its library, and the library has to be refreshed. How that happens depends on your tool.

Two copies, and only one is read

When you add a document, the tool takes a copy, splits it up and builds an index. Those steps are described in this explainer on chunks. The assistant searches the copy. Your original is a separate file.

AWS asks the question directly: what if the external data becomes stale? Its answer is to asynchronously update the documents and update the embedding representation of the documents.¹ An embedding is the number list that stands for a chunk's meaning, which the embeddings page explains. So a changed file has to be read again, and its numbers recalculated.

How the refresh can happen

AWS says this can be done through automated real-time processes or periodic batch processing.¹ Plainly put:

• Automatic and quick. The tool notices a change and updates the library soon after.
• Automatic on a schedule. The tool checks at set times, so there's a gap between your edit and the new answers.
• By hand. You re-upload, or press a "sync" or "refresh" button.

Microsoft's documentation says incremental indexing keeps content fresh, and it advises you to review indexing concepts and strategies to determine how you want to ingest and refresh data.² Those pages are for developers. As a business owner, the useful question to ask of any product is: how does a change reach the library, and how long does it take?

What you may see after a change

You edited a file. Until the refresh finishes, the assistant may still answer from the old text. Wikipedia says new information doesn't call for retraining the model. The knowledge base is the part that gets the update.³ The model isn't the problem. The library is.

You uploaded a new version without removing the old one. Both may be searchable. In the worst case, says Wikipedia, answers merge outdated and updated information in a misleading manner.³ Replace the old file, or delete it.

You deleted a file. Check the tool's documentation on whether the copy and its index entries go too. After deleting, ask a question only that file could answer. If the assistant still answers it, the old copy is probably still there.

You renamed or moved a file. The documentation for your tool should say what happens. If it doesn't, ask a test question after any move.

A routine that suits a small team

• Give each document an owner who is responsible for it.
• Write the date of the last review inside the document, near the top. Staff can then spot an old file at a glance.
• Review the documents that carry prices, hours, fees and policies on a regular schedule that you choose.
• After any change, ask the assistant a question only the new text can answer.
• Keep a short list of test questions with known answers. Run them after big edits.

None of this needs special software. A shared spreadsheet listing each file, its owner and its last review date is enough.

When an answer looks out of date, reading the source links shows which file it came from. That helps you work out whether the library or the lookup is behind.