How to build an AI assistant for your company knowledge base
A knowledge assistant is useful when it answers from trusted internal sources, shows where each answer came from and says so when the knowledge base has no answer. This article explains, for SME owners and operations leads, how such an assistant works, what its source documents need and how to keep the answers reliable after launch.
Many businesses carry important knowledge in scattered documents, inboxes, spreadsheets and people's heads. The current price list is in one folder, and the answer a customer received last year is in a colleague's mailbox. AI can make that knowledge easier to reach, provided the assistant is connected to trusted sources and built around the way staff ask for help.
How an AI knowledge base assistant works
A language model has broad general knowledge, but it has not read your price list or your service manual. Asked about them directly, it will still produce a fluent answer, and nothing on the screen shows whether that answer is right.
Retrieval is the usual way around this. The assistant looks up the relevant passages in your documents each time a question is asked and writes its answer from those passages, without relying on what the model remembers. The technical name is retrieval-augmented generation, usually shortened to RAG.
It works much like a careful new colleague who checks the handbook before replying and tells you which page the answer is on. Each question passes through four steps.
- A member of staff asks a question in their own words.
- The system searches the connected sources and picks the passages that best match the question.
- The model writes an answer using those passages.
- The answer appears with links to the documents it was drawn from.
Because the answer follows the documents, a corrected document leads to a corrected answer with no change to the model. Each answer can also be checked against its source, which makes mistakes easier to find and fix.
Start with the questions your team asks
Begin with one team and the questions it asks again and again. Collect them from the shared inbox and from the colleagues everyone turns to. For each question, find the document or record that holds the answer and note who maintains it. Block Consult's company knowledge and AI integration work starts the same way, with the tasks, the sources behind them and the owners of those sources.
Take a service team that keeps asking which warranty terms apply to an older product line. If the answer sits in a PDF that two people know about, that question and that PDF belong in the first version. If no document holds the answer, you have found the first gap to close before any software is involved.
Prepare the source documents
A useful assistant needs source material that is current, clearly named, permissioned and structured enough to retrieve. Old policies and duplicate templates produce confused answers, because the assistant cannot tell which version the company stands behind.
Work through the sources for the chosen questions. Archive superseded versions, remove duplicates, give files names that say what they contain and record an owner for each source. Scanned pages and tables may need converting before their text can be searched. Our guide to preparing documents and data for an AI workflow sets out the checks in full.
Show the source with every answer
The assistant should show source links or document names with its answers, above all where a decision depends on them. A colleague can then open the passage and check it, and the team learns when to rely on the assistant and when to look more closely.
Decide what happens when retrieval finds nothing suitable. The assistant should say that it has no source for the question and point to a person who can help.
Carry permissions through to the assistant
Separate public, internal and restricted material before connecting it. The assistant should answer only from documents the person asking is allowed to open, so a question about salaries returns nothing to someone outside HR. That takes permissions in three places: the document sources, the retrieval step and the access rules of the application itself.
Define which topics the assistant should refuse or pass to a person, and test access with accounts from different roles before launch. The rules for confidential data and sensitive topics belong in the company's AI governance, so that they apply to every tool the team uses.
Test with common questions before launch
Run the collected questions through the assistant with the people who will use it. Ask them to mark each answer as correct, wrong or missing a source. Fix the documents first, then adjust how retrieval is set up, and keep the questions as a test set to repeat whenever sources or settings change.
An assistant that lives in a separate tab is easily forgotten. Placing it inside the tools the team has open all day is an integration task, described in our article on connecting AI to the software your team uses.
Measure gaps as well as answers
Unanswered questions reveal missing documentation, unclear processes and training needs. Give users a simple way to flag a wrong or missing answer, and record every question the assistant could not answer.
Someone has to act on that record. Name an owner who reviews it at a fixed interval and passes each gap to the person responsible for the source. The record then doubles as a to-do list for the company's documentation, ordered by what staff have asked for.
Frequently asked questions
- What is RAG in an internal AI assistant?
- RAG stands for retrieval-augmented generation. Before answering, the assistant searches your internal sources for relevant passages and writes its answer from them.
- Can a knowledge assistant respect permissions?
- Yes, if permissions are designed into the document sources, the retrieval step and the application's access rules. The assistant should only answer from documents the person asking is allowed to open.
- How do we improve answer quality?
- Improve the source documents first by removing duplicates and archiving old versions. Then add a feedback path, tune retrieval and test common questions with the teams that will use the assistant.
- Does the assistant need to be trained on our documents?
- Usually not. With retrieval, the documents stay where they are and the assistant looks them up for each question, so the answers follow the current versions.
Related reading
Let's build something durable.
Tell us about your project. We reply within two business days.

