Govern the source material before choosing a model
Policies, manuals, project files and support scripts often contain duplicate versions, outdated content and different access levels. Start by defining ownership, validity, update time and scope.
If source material conflicts, even accurate retrieval cannot reliably determine which document is authoritative.
- Catalog material by department, workflow and confidentiality
- Retain version, publication date, expiry date and owner
- Remove scan noise, duplicate headers and irrelevant attachments
Retrieval quality sets the ceiling for answer quality
Chunk size, heading hierarchy, table handling and the combination of keyword and vector retrieval all affect recall. A few demo questions are not enough; use real business questions as a repeatable evaluation set.
Evaluation should cover source recall, factual consistency, accessible citations and refusal behavior—not merely fluent wording.
Permissions must apply before retrieval
An enterprise knowledge base should not retrieve everything and then ask the model not to disclose it. Identity, department, project and record-level permissions must filter retrieval, with access and citation logs retained.
- Separate public, internal and sensitive material
- Reuse existing identity and organization permissions
- Add confirmation for export, sharing and high-risk answers