10.5 Responsible AI and Security Review Board Lab
Key Takeaways
- A review board should ask practical questions about fairness, explainability, privacy, safety, transparency, accountability, and human oversight without pretending every project needs the same control set.
- Security review must cover IAM least privilege, encryption, data classification, prompt injection, logging, retention, monitoring, and shared responsibility.
- AWS services such as Guardrails for Amazon Bedrock, SageMaker Clarify, Amazon A2I, Macie, CloudTrail, CloudWatch, AWS Config, Audit Manager, Artifact, KMS, and Secrets Manager support different review needs.
- AIF-C01 practitioners should know what evidence to request and when to escalate, not implement full governance frameworks or custom security protocols.
- The board decision should be documented as approve, approve with conditions, defer, or reject, with owners and review dates.
Lab scenario: review board for AI launch approval
A company has three AI proposals ready for review: a support assistant, a document summarizer, and a churn-risk dashboard. The review board includes product, security, privacy, legal, compliance, operations, and customer support leaders. The board is not there to write code or tune models. Its job is to decide whether each use case has enough evidence, control, ownership, and monitoring to move forward responsibly.
At the practitioner level, responsible AI is a set of concrete questions. Is the use case appropriate for AI? Could the output harm a person, customer, employee, or regulated decision? Is the data permitted for this use? Can users tell when AI is involved? Is there a human review path? Are errors monitored? Who owns remediation? The board should avoid both extremes: approving everything because a managed service is used, or blocking every project because AI has risk.
| Review area | Evidence to request | AWS-related control examples |
|---|---|---|
| Use-case fit | Business goal, non-AI baseline, risk tier, user impact | Well-Architected review questions and documented architecture. |
| Fairness and bias | Affected user groups, labels, evaluation slices, complaint path | SageMaker Clarify for applicable ML workflows, business review of model outcomes. |
| Explainability | Reason codes, citations, feature importance, reviewer notes | RAG citations, model evaluation records, Clarify where relevant. |
| Privacy | Data inventory, consent or lawful basis, retention, redaction | Macie, KMS, S3 bucket policies, IAM, log retention controls. |
| Safety | Unsafe content, prompt attacks, denied topics, human handoff | Guardrails for Amazon Bedrock and application escalation rules. |
| Accountability | Decision owner, approval owner, incident owner, review cadence | Ticketing records, CloudTrail, CloudWatch, Audit Manager evidence. |
| Security | Least privilege, secrets, encryption, network path, monitoring | IAM, KMS, Secrets Manager, CloudTrail, AWS Config, Inspector, Trusted Advisor. |
The board should classify each proposal by impact. A low-risk internal summarizer of public release notes may need light review. A model that recommends account closure, credit denial, medical advice, or employee discipline needs deeper review, legal analysis, human oversight, and likely should not be automated casually. The AIF-C01 candidate is not expected to build a governance framework, but should recognize that risk level drives controls.
Responsible AI evidence should be specific. For the support assistant, ask for the prompt template, source corpus, retrieval tests, guardrail settings, sample bad-answer review, and human approval workflow. For the document summarizer, ask for extraction confidence thresholds, reviewer authority, sensitive data handling, and audit evidence. For the churn dashboard, ask which features are used, whether protected or proxy attributes are present, how false positives are handled, and whether customers or employees are affected by the prediction.
Security review starts with identity and data. Use IAM least privilege for application roles and service roles. Store secrets in AWS Secrets Manager, not in prompts or code. Encrypt data at rest with KMS where appropriate and protect data in transit. Decide whether prompt and response logs are stored, what they may contain, who can read them, and how long they are retained. Use CloudTrail for API activity, CloudWatch for operational metrics, and AWS Config or Audit Manager where the organization needs configuration or compliance evidence.
Prompt injection belongs on the board agenda. A malicious user or a document inside a RAG corpus can tell a model to ignore instructions, reveal hidden prompts, call tools, or leak data. Controls include Guardrails, strong prompt boundaries, retrieval source governance, narrow action schemas, least-privilege tool roles, and human approval for sensitive actions. The board should ask for adversarial test examples, not only positive demos.
Decision log template:
- Proposal name and business owner.
- AI function: classify, recommend, generate, summarize, retrieve, forecast, or automate.
- Risk tier and reason.
- Data sources, data classes, retention, and access boundaries.
- Human review point and escalation owner.
- Evaluation evidence, including normal cases, edge cases, and unacceptable outputs.
- Security controls, logging, monitoring, and incident response path.
- Board decision: approve, approve with conditions, defer, or reject.
- Re-review trigger: new data source, new model, new audience, new action capability, or incident.
Failure modes should change the decision. If nobody owns source content, defer the RAG assistant. If customer data appears in unprotected logs, reject production use until logging is corrected. If a churn model uses a proxy for a sensitive attribute, require fairness review before launch. If a document summarizer cannot show source evidence for generated conclusions, keep it in assistive review mode. If an agent can call broad administrative APIs, narrow the action group before approval.
Review prompts before the quiz:
- Which proposal has the highest user harm if the AI is wrong?
- What proof shows that human reviewers can override the AI?
- Which logs or prompts might contain sensitive data?
- Which data source, model, or audience change should force re-review?
- What condition would make the board reject rather than approve with conditions?
A review board is evaluating a RAG assistant that uses employee documents. Which evidence is most important before approval?
A generative AI app stores prompts and responses for troubleshooting. What security and privacy question should be asked?
Which board decision is most appropriate when a high-impact AI workflow lacks a human appeal or override path?