5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)

What guardrails do

Guardrails are runtime controls that constrain what goes into and comes out of a generative model, reducing the risk of harmful, non-compliant, or malformed behaviour. Unlike offline evaluation, guardrails operate in production on live traffic. They complement testing rather than replace it: testing tells you how the system behaves, while guardrails enforce limits when it misbehaves. From a testing standpoint, guardrails themselves must be tested — for both false negatives (unsafe content allowed through) and false positives (safe content wrongly blocked).

Risks guardrails address

Generative systems fail in characteristic ways that guardrails are designed to contain:

• Prompt injection and jailbreaks — crafted inputs that override system instructions or bypass safety rules.
• Harmful or toxic content — hate, harassment, self-harm, or violent output reaching users.
• Sensitive-data leakage — exposing PII, secrets, or proprietary context in a response.
• Hallucination and non-compliance — confident but false claims, or answers that violate legal or policy constraints.
• Malformed output — responses that break the schema a downstream system expects.

No single control covers all of these, which is why guardrails are layered and combined with evaluation and monitoring rather than relied on alone.

Types of guardrail

Guardrails act at three points: on the input, on the output, and on the conversation flow.

Input filtering inspects the incoming prompt to catch prompt-injection or jailbreak patterns, disallowed topics, or sensitive data before the model ever sees them. Output filtering inspects the generated response, redacting PII or secrets and blocking policy-violating text before it reaches the user. Content moderation applies classifiers for categories such as hate, harassment, self-harm, sexual content, and violence, on either side. Schema validation is a deterministic guardrail: for structured outputs, the response is validated against a JSON schema or type contract, and non-conforming outputs are rejected or repaired — one of the most reliable guardrails precisely because it is not itself probabilistic.

Representative tooling

• Llama Guard is a safety classifier model. It takes a prompt or a response and classifies it as safe or unsafe against a defined taxonomy of hazard categories, returning which category was violated. It is used as an input and/or output moderation check and can be aligned to a policy taxonomy. Because it is itself a model, it must be evaluated like any classifier — measuring precision and recall on labelled safe/unsafe examples — and it will make mistakes on adversarial or ambiguous inputs.
• NeMo Guardrails is a programmable framework for adding rails to LLM applications. Using a rail specification (dialog and flow definitions, for example in a Colang-style format), it can enforce topical boundaries, define allowed dialog flows, call moderation checks, and orchestrate several rails around a conversation. It can chain input rails, dialog rails, and output rails, giving programmatic control over what the assistant may discuss and how it responds. Its emphasis is defining and controlling dialog behaviour, not only classifying a single message.

The distinction matters for the exam: Llama Guard is a classifier applied at a point in the pipeline, whereas NeMo Guardrails is an orchestration layer that can invoke classifiers and enforce conversational rules.

Benchmarks as evaluation resources

Alongside runtime guardrails, testers use safety and truthfulness benchmarks to evaluate models before and after deployment.

• TruthfulQA measures a model's tendency to reproduce common human misconceptions — a truthfulness benchmark, not a runtime filter.
• Other public benchmarks cover toxicity, bias, and refusal behaviour.

These benchmarks are evaluation datasets, not guardrails; they help quantify risk so that appropriate runtime guardrails can be specified and prioritised.

Testing the guardrails

Because guardrails are themselves software with safety impact, they need their own test strategy:

• Adversarial / red-team inputs — jailbreaks, injections, and obfuscated harmful requests — to probe false negatives.
• Benign edge cases — legitimate medical, security, or historical questions — to measure false positives and over-blocking.
• Measuring both error types — track the trade-off, since overly strict rails harm usefulness while lax rails harm safety.
• Layering / defence in depth — combine deterministic schema checks with classifier-based and rail-based controls, because no single guardrail is complete.

Placement and monitoring

Guardrails sit at defined points in the request pipeline: pre-model checks screen the user input, post-model checks screen the response, and flow-level rails govern the whole conversation. In production they should log every block-or-allow decision so testers can audit them, measure false-positive and false-negative rates over real traffic, and tune thresholds. Because attackers adapt, guardrail rules and classifier taxonomies need periodic review, and monitoring dashboards should track blocked-request trends as an ongoing signal rather than a one-time gate.

Guardrails reduce but do not eliminate risk; they are one layer in a defence-in-depth approach that also includes prompt design, evaluation, monitoring, and human oversight.