10.4 Enterprise Search, RAG, and Knowledge Management Lab

Lab scenario: enterprise knowledge assistant

A growing company has policy PDFs in S3, engineering runbooks in a wiki, support tickets in a case system, onboarding material in shared drives, and old process documents in several folders. Employees ask the same questions repeatedly because they do not know which source is current. Leadership wants an internal assistant that answers questions, cites sources, respects permissions, and helps knowledge owners find stale content. The pilot audience is employees, not external customers.

The first design question is whether the organization needs search, RAG, or a broader managed assistant. Search returns ranked documents or passages. RAG retrieves relevant context and asks a model to answer from it. A managed enterprise assistant can add connectors, conversational experience, and permission-aware knowledge access. These options overlap, but the correct choice depends on the user experience, source systems, answer style, security model, and operations team.

Start with a source inventory. List each repository, owner, content type, audience, update cadence, classification, retention rule, and access model. Mark which sources are authoritative and which are legacy. A common failure mode is indexing everything because it is easy, then watching the assistant answer from old onboarding decks or draft legal text. A knowledge assistant is only as trustworthy as the content lifecycle behind it.

Permissions need direct testing. If the HR policy folder includes compensation documents, the assistant must not expose those documents to all employees. If engineering runbooks include incident response steps, access may depend on team or role. Permission enforcement can use the managed service capability, source-system permissions, IAM, application authorization, metadata filters, source partitioning, or separate corpora. The lab should include two users with different access and the same question. The expected result is different retrieved content where permissions differ.

RAG design should include metadata and citation rules. Metadata such as department, region, product, effective date, source owner, document status, and audience helps retrieval choose the right answer. Citations help users verify. The prompt should instruct the model to answer only from retrieved sources, include source links where available, and state when the answer is not found. It should not turn one weak passage into a confident enterprise policy.

Prompt injection is an enterprise search problem, not only a chatbot problem. A ticket, wiki page, or document can contain text telling the model to ignore instructions, reveal hidden prompts, or exfiltrate data. Retrieved content should be treated as data, not as higher-priority instructions. Guardrails, prompt design, tool limits, source sanitation, and least privilege all help. If the assistant can take actions, action permissions must be separate from search permissions.

Decision log example:

• Pilot corpus: HR public policies, IT onboarding guides, and product support runbooks approved by their owners.
• Excluded corpus: draft legal documents, customer contracts, employee records, and old migration folders.
• Service path: Amazon Q Business for managed employee assistant, or Bedrock Knowledge Bases if the team needs a custom app experience.
• Freshness rule: content owner must update or retire documents; indexes refresh on a defined schedule.
• Permission test: employee, manager, HR partner, and engineer roles each run the same set of questions.
• Bad-answer process: users can flag answers, and owners review retrieval source, model response, and corpus quality.

Failure modes to test:

• The assistant answers from a superseded policy because effective date metadata is absent.
• The assistant retrieves a document the user should not see.
• The assistant gives a generated answer without citations for a policy question.
• Conflicting sources exist and the answer hides the conflict.
• A wiki page contains prompt injection text and the assistant follows it.
• A source connector fails silently and employees rely on incomplete answers.
• The answer is correct for one country but wrong for another.

Review prompts before the quiz:

• Which source is authoritative when two documents conflict?
• Who owns stale content removal, not just assistant tuning?
• Which user roles must be tested for permission boundaries?
• Where should the assistant return search results instead of a generated answer?
• What operational metric shows knowledge management is improving: fewer repeat tickets, faster onboarding, higher search success, or fewer flagged answers?