How to test an AI knowledge assistant before launch

A knowledge assistant should answer from the right documents, show evidence and respect the user's access permissions. Testing those properties separately reveals problems that a fluent answer can hide, and gives the team a clear basis for deciding whether to launch.

Block Consult3 min read
In this article
Figure · Knowledge Systems

Turn real questions into a test set

Ask the intended users for questions from their normal work. For each one, record the relevant source, its effective date, the user's role and the facts a satisfactory answer must contain. Have the document owner confirm the expected answer before testing. Otherwise an outdated answer key can make the correct output look wrong.

Include questions that should not receive a substantive answer. Some will have no approved source; others will ask for information outside the user's access. Keep a portion of the cases out of tuning and use them for the final review. This extends the preparation in our guide to building a company knowledge assistant.

Check retrieval, answer and evidence separately

  • Retrieval: did the system find the current, relevant passage that this user is allowed to read?
  • Answer: are the important facts correct, and are conditions or exceptions preserved?
  • Evidence: does each material claim follow from the linked passage, and does the link open the intended version?
  • Abstention: when evidence is absent or conflicting, does the assistant identify the gap and give an appropriate next step?

A citation is useful only if it supports the answer. For example, a response about training approval might cite a real handbook but omit its requirement for manager approval. Record that as an answer failure even though retrieval succeeded. If the current handbook was never found, investigate the source and search process first.

Keep the outcomes separate in the report. Track accepted answers among answerable cases, appropriate abstentions among unanswerable cases and permission failures across access tests. One overall percentage makes it too easy for routine questions to conceal a serious failure in a smaller group.

Test the same question under different permissions

Create test identities with different roles and repeat the same questions. Authorisation needs to apply to retrieval before restricted material reaches the model. Microsoft's document-level access guidance describes filtering search results by permissions. A prompt asking the model to keep a secret is not an access-control boundary.

Then change a role or revoke access and test again. Check cached results, conversation history and source links as well as the new answer. Record the expected time for permission changes to take effect. The product owner should know whether the assistant temporarily retains access that the source system has already removed.

Include documents that challenge the assistant

Use a small set of deliberately difficult test documents: an obsolete policy beside its replacement, a table whose meaning depends on a footnote and two sources with conflicting instructions. Ask for the effective policy and inspect the passages the assistant used. When the sources cannot resolve the conflict, the expected result should identify that uncertainty.

Also include a test document containing instructions directed at the assistant, rather than information for the reader. It should be treated as source content, not authority to change the task or access rules. OWASP's guidance on retrieval weaknesses discusses poisoned source material and information leaks between access groups.

Define release criteria and repeat the checks

Agree release criteria for each group of cases before the final run. Permission leaks or fabricated evidence should block release until investigated and corrected. Set quality expectations with the people doing the work, and record remaining limitations in the user guidance. A small evaluation set is a starting point; add new cases from observed failures.

Keep the model, prompts, retrieval settings and source snapshot with the results. Repeat relevant checks when any of them changes, including when a policy is replaced or a connector gains access to another document collection. The process depends on maintained source data and belongs in the ongoing care of the assistant.

Frequently asked questions

Is an answer reliable if it includes a citation?
Only if the cited source supports the material claims and is the appropriate version. A genuine link can accompany an incorrect interpretation, an omitted condition or an outdated answer.
Should every unanswered question count as a failure?
No. For questions without an approved source or the required access, a clear abstention can be the correct result. Measure those cases separately from questions the assistant should answer.
Who should approve the evaluation results?
The workflow owner should make the release decision, supported by document owners for factual accuracy and the responsible access or security owner for permission tests. Reviewers need enough context to judge the actual task.
Contact

Let's build something durable.

Tell us about your project. We reply within two business days.

EssentialAs needed
We use storage to keep portal sign-in secure, verify invitations and remember that you’ve seen this notice. Security controls also help protect the site.
Your preferences and toolsWhen you use them
Your browser can save your chosen theme, chat conversation in this tab, simulator model and workspace preferences. You can clear this data in your browser settings.
Analytics and advertisingNot used
We don’t use Google Analytics, advertising pixels or cross-site tracking. We use Search Console to see how the site appears in Google Search; it adds no tracking script to this site.

Acknowledging this notice does not give consent to tracking. We will ask for your consent before adding optional tracking.

Cookie policyPrivacy policy