Answer in brief
Treat the knowledge base as a set of maintained documents rather than as a clever model. Give every document one named owner and a review-by date, replace instead of adding when facts change, tell the assistant to cite its source and to say when it does not know, limit retrieval by role, and rerun a fixed list of test questions after every update.
A stale file does more damage than a weak model
An assistant that answers from your own documents does not know which of them is current. It finds the passages that best match the question and writes a fluent reply around them. If last spring’s price list still sits in the folder beside this week’s, either can come back, and the reply will sound equally sure in both cases.
This arrangement has a name, retrieval-augmented generation. Amazon Web Services describes it as a way of making a language model consult a trusted body of knowledge that lies outside its training data before it answers. The same overview lists what such models do badly without that help, among them serving stale information to someone who wanted a precise, up-to-date reply. Retrieval repairs that only when the documents themselves are current.
That is good news for a small company. AI knowledge base maintenance is mostly housekeeping that any owner or office manager already understands, not data science. The questions are plain ones: who answers for each document, when it stops being true, how it is replaced, who may read it and how the replies are checked. The rest of this article takes them in turn.
One named owner for every document in the base
Every file the assistant can read needs one person who answers for it. Not a department and not “the team”: a name. The owner confirms that the content is true today, changes it when the facts change and removes it when it is no longer needed. Without that name, an outdated delivery policy belongs to nobody and stays in the index for a year.
The second rule is one home for each fact. The standard price of a service lives in the price list and nowhere else; the proposal template and the onboarding guide refer to it instead of repeating the figure. When the same number is typed into five documents, one of them will be missed at the next change, and retrieval has no way of telling the forgotten copy from the corrected ones.
Keep the result in a register, which can be an ordinary spreadsheet: document, owner, where the facts come from, date last confirmed, review-by date and who is allowed to read it. The Generative AI Profile published by the US National Institute of Standards and Technology (NIST AI 600-1) asks for the same thing in institutional language: roles and responsibilities clearly defined, including the frequency of periodic review. The framework is voluntary, and a one-page register is a proportionate way for a firm of ten people to follow it.
Expiry dates turn old facts into visible ones
A document without a date is assumed to be true for ever. Give each one a review-by date that matches how fast its subject moves. A price list in a season of supplier increases may need confirming every week; delivery terms, whenever the carrier changes them; a returns policy, once a quarter; the company history, once a year. The intervals are yours to choose. What matters is that each one exists and is written down.
Put the date in two places. Inside the document, as a plain line such as “valid from 1 March, review by 31 March”, so that the assistant can repeat it in its reply. And in the register or the file’s properties, so that an expired document can be left out of retrieval or flagged until its owner confirms it again. NIST’s profile quotes a definition of high-integrity information that suits a price list exactly: it “creates reasonable expectations about when its validity may expire”.
Some facts go out of date too quickly for any document. Stock levels, today’s exchange rate and free appointment slots change by the hour, and a file that copies them is wrong almost as soon as it is saved. Either connect the assistant to the system that holds the live figure, or instruct it to say where that figure is kept and who can confirm it. An honest pointer is more useful than a precise number from last Tuesday.
The weekly update routine: replace, remove, re-index
AI knowledge base maintenance fails when it depends on someone remembering. Fix a day and a short routine. Each owner looks at what changed during the week, whether a supplier’s price, a policy, the opening hours or a discontinued product, and edits the document that holds that fact. Then the old version leaves the indexed folder altogether. Adding the new file next to the old one is the commonest way of creating two truths.
The index has to keep up. AWS’s overview raises the problem of external data going stale and answers that both the documents and the stored representations built from them have to be refreshed, whether continuously by an automated process or in batches at set intervals. Microsoft’s documentation for its search service makes the same point when it lists incremental indexing as the means of keeping content fresh. Whichever product you use, find out when it re-reads your files and whether deleted files really disappear from its replies.
Hunt for duplicates as part of the routine. Old PDFs on a shared drive, a price list attached to a chat last year and files called “final-v2” are all candidates for an answer if the tool can reach them. Archive them outside the folders the assistant reads. Finish by adding one line to a change log: what changed, who changed it and on which date. When a reply is questioned later, that log is the quickest way to find out why.
Teach the assistant to say it does not know
The dangerous reply is not the refusal but the confident guess. NIST calls it confabulation: the production of confidently stated but erroneous or false content, known colloquially as hallucination. AWS reaches for a more domestic picture and compares a language model to a keen new recruit who never keeps up with events yet replies to every question with complete assurance. A knowledge base reduces the guessing only if the assistant is told what to do when the search comes back empty.
Write the rule into its instructions in plain words. Answer only from the retrieved documents. If none of them covers the question, or the one that does has passed its review-by date, say so and name the person to ask. Never work out a price, a discount or a deadline from general knowledge. On matters such as contracts, staff disputes or refunds outside the written policy, pass the question to a person rather than attempt an answer.
Ask for the source every time. A reply that ends with the document’s name and its date can be checked in ten seconds, and AWS notes that output with citations lets users look up the source documents for themselves. NIST adds a caution worth remembering: a model can also produce invented citations that appear to justify its answer. The reference must therefore be a link to a real file in your base, not a title the model has composed.
Access rights come before convenience
An assistant that can read everything will sooner or later tell someone something they were not meant to see. Microsoft’s documentation gives the textbook case: finance data, which ought to stay with the finance team even if a senior executive puts the question to the chatbot. It states the requirement briefly, that people and software agents should be able to retrieve only what they are authorised to see, and lists document-level security trimming, in which results are filtered by the permissions of the person asking, among the controls.
A small firm rarely needs elaborate roles. Three or four audiences are usually enough: all staff, managers, the people who handle finance and personnel records, and the owner. Sort the documents into folders or collections on that basis and connect each group of users only to its own. Check whether your tool applies the permissions of the original file store or needs them to be set again; the two behave differently and the difference is easy to miss.
Decide what never goes in. Payroll, contracts with named individuals, medical notes, customer complaints containing personal details and passwords have no place in a general knowledge base, however convenient a quick answer would be. If customers are to use an assistant as well, give it a separate and much smaller base holding only what you would publish on your website. Remove a person’s access on the day they leave, as you do with email.
A fixed list of test questions, rerun after each change
Trying three questions and feeling pleased is not a test. NIST’s profile says as much in its own way: it advises against extrapolating a system’s performance from narrow, non-systematic and anecdotal assessments, and recommends comparing output with a set of known ground-truth data. For a small business that means a written list of questions, each with the correct answer, the document it should come from and the date that document carries.
Build the list from real life: the questions staff asked last month, taken from chat and email. Twenty or thirty are enough to begin with. Add three kinds of trap. The first is a question whose answer changed recently, which catches an old price. The second is a question the base cannot answer, which shows whether the assistant declines. The third is a question about a restricted document asked from an ordinary account, which tests the permissions.
Rerun the whole list after every update of the documents and after any change of model, settings or instructions, since each of these can alter replies that used to be right. The profile recommends re-assessing risks once retrieval has been introduced, and reviewing sources and citations both before deployment and during ongoing monitoring. Mark each reply on three points, whether the fact, the source and the date are correct, and do not release the update until the failures have been explained.
Spotting drift and knowing when to pause the assistant
Between test runs, the cheapest monitor is the people who use the assistant. Give them one obvious way to report a wrong answer, whether a button, a shared channel or an email address, and make sure every report reaches the owner of the document concerned. NIST lists this among its suggested actions: evaluate the problematic content that users report and integrate their feedback into updates. Each report becomes a new line in the test list.
Read the refusals too. A log of the questions that received “I do not know” is a map of what the knowledge base lacks, and it tells you which document to write next. Count the expired documents every week; a rising number means the routine has stopped working, whatever the replies look like. Review a few real replies from time to time, because a system that is usually right is exactly the one people stop checking. NIST names the habit automation bias: excessive deference to automated systems.
Keep the right to switch a topic off. During a price change, a stock count or the rewriting of a policy, it is safer for the assistant to say that the information is being updated and to give a contact than to answer from documents in transition. Start with one narrow area in which the facts are well kept, such as prices and delivery, and widen the scope only when the register, the routine and the tests have run without incident for a few weeks. An AI knowledge base for small business succeeds as a maintenance habit long before it succeeds as technology.
Practical checklist
- List every document the assistant can read and put one owner’s name and a review-by date beside each of them.
- Keep each fact in a single document, and take the previous version out of the indexed folder whenever you replace it.
- Write the refusal rule into the assistant’s instructions: no source means no answer, plus the name of a person to ask.
- Sort documents by audience, connect each group of users only to its own folders and leave payroll and personal records out.
- Maintain a list of test questions with expected answers, sources and dates, and run it after every update.
Questions and answers
What is an AI knowledge base in a small company?
It is a set of the company’s own documents that an AI assistant searches before it replies, so that the answer comes from your price list or policy rather than from the model’s general training. The technique is called retrieval-augmented generation. The assistant is only as accurate as those documents, which is why ownership, dates and updates matter more than the software.
How often should the documents behind an AI assistant be updated?
As often as the facts in them change, which differs from one document to another. Prices and product availability may need a weekly check, policies a quarterly one and background material an annual one. Set a review-by date for each document, confirm or replace it on that date, and delete the old version from the indexed folder at the same time.
What should the assistant do when the answer is not in the documents?
It should say that it has no source for the question and tell the person whom to ask. This behaviour has to be written into its instructions, because a language model left to itself tends to produce a plausible reply instead. Check it regularly with questions you know the base cannot answer, and treat any invented reply as a defect to be fixed before wider use.
Can employees with different roles share one internal assistant?
Yes, provided retrieval respects permissions. Each person should receive answers only from documents they are entitled to open, which in practice means sorting files by audience and confirming how your tool applies access rights. Sensitive records such as payroll and personal data are better left out entirely. Test the arrangement by asking for a restricted document from an ordinary account.
How can a non-technical owner check that the answers are right?
With a written list of real questions and the answers you expect, including the document and date each should cite. Ask them all after every change and compare the replies by hand; no special software is needed for a list of a few dozen. Include questions that ought to be refused. If an answer is wrong, find out whether the document, the index or the instruction is at fault before changing anything else.