6.4 AI Access, Data, Input, Output, and Audit Controls

6.4 AI Access, Data, Input, Output, and Audit Controls

Securing an AI service requires more than filtering a chat box. The service may expose multiple modalities, retrieve confidential records, invoke agents and plug-ins, and pass generated output into high-impact workflows. CY0-001 therefore expects controls at every trust boundary.

Identity, agent, and endpoint access

Authenticate people, workloads, and service identities separately. Enforce least privilege with role- or attribute-based access control, short-lived credentials, managed secrets, and separation between development and production. Agent access control requires explicit authorization for every tool and resource; the fact that a user may ask a question does not imply that the model may send email, modify a ticket, query payroll, or run code. Use allowlisted actions, scoped tokens, transaction limits, approval gates, and a kill switch.

Inventory every API, model, plug-in, vector store, and orchestration endpoint. Restrict network paths, validate tenant context on the server, and rate-limit both users and machine identities. An input quota limits request count, token count, file size, retrieval depth, tool calls, or spending within an interval. Quotas help contain denial of service and unbounded consumption, but limits must cover downstream calls too; a short prompt may trigger an expensive agent loop.

Modality controls address text, image, audio, video, and files. Check declared and actual file type, size, archive depth, malware, optical text, metadata, and hidden instructions. Disable unused modalities. Treat text recovered by OCR and speech-to-text as untrusted input, not as trusted operator instruction.

Data protection

Begin with classification labels and approved-purpose rules. Minimize collected fields and tokenize or redact credentials, secrets, personal data, and regulated identifiers before they enter prompts or logs. Anonymization aims to prevent identification; pseudonymization merely replaces direct identifiers and remains linkable. Test either approach against re-identification and embedding-inversion threats.

Apply encryption in transit and at rest to datasets, vector indexes, artifacts, caches, prompts, outputs, and backups. Where the platform and threat model justify it, confidential-computing technology can provide encryption in use or isolation, but it does not make a malicious prompt or incorrect model output safe. Control keys separately, rotate them, and prevent an inference service from exporting raw model weights.

Input and prompt controls

A prompt firewall inspects prompts and retrieved content for injection patterns, encoded payloads, unsafe instructions, excessive length, and policy violations. Input validation checks type, range, encoding, origin, and business rules. Sanitization can normalize Unicode, strip active markup, remove control characters, and isolate quoted data. These controls reduce risk but have false positives and evasions, so do not rely on phrase matching alone.

Separate instructions from data with explicit structures. Sign trusted templates, version the system prompt, and mark retrieved content as untrusted. Prevent user input from selecting arbitrary tools, destinations, or data scopes. Protect prompts as configuration: restrict access, review changes, and do not log hidden instructions or secrets unnecessarily.

Output and action controls

Validate output before display, storage, or execution. Use schemas and allowlists for structured fields, escape content for the destination context, scan for secrets and prohibited data, verify factual claims against authoritative sources, and enforce tenant authorization again. Confidence is useful for routing but is not proof of correctness; calibrate it on representative data and combine it with impact. A low-impact suggestion may be shown with a warning, while a destructive action requires human approval regardless of model confidence.

Improper output handling occurs when generated text becomes SQL, shell, HTML, code, or a tool argument without destination-aware validation. Keep the model advisory by default. Sandboxed execution, read-only access, transaction previews, dual authorization, and rollback are stronger controls than asking the model to be careful.

Monitoring and auditability

Use prompt monitoring and log monitoring to record prompts, sanitized variants, retrieval references, outputs, policy decisions, tool requests, token and monetary cost, latency, confidence, errors, abuse signals, and final outcomes. Protect monitoring data because logs can contain the same secrets as prompts. Use access controls, integrity protection, retention limits, and redaction.

A reconstruction-quality audit event links timestamp; actor and tenant; session and request IDs; model, dataset, system-prompt, and policy versions; retrieved source identifiers; input and output hashes or properly protected content; tool calls; quota decisions; human approvals; and disposition. It should answer who caused what system to use which evidence and authority. Audit trails support incident response and compliance, but collecting every raw prompt forever can increase privacy risk.

Scenario

An agent drafts firewall changes from incident tickets. The safe design gives it read-only ticket access, a schema-limited rule proposal tool, and no direct firewall credential. Uploaded files are scanned and OCR text is isolated as data. The prompt firewall flags manipulation attempts, while server-side authorization prevents cross-tenant retrieval even if the filter misses one. A deterministic validator rejects broad CIDR ranges, a human approves the preview, and a separate deployment identity applies it. Logs preserve versions, evidence, approval, and result without retaining an exposed API key. This layered design remains safe when any single control fails.