What is a guardrail?

What a guardrail actually does

A guardrail is a separate check that runs before or after the model call and decides whether the turn should proceed. The input version reads the user message and any retrieved context, then flags anything the model should not be asked to answer: policy violations, prompt-injection payloads, off-topic questions, personally identifying data. The output version reads whatever the model produced and flags anything the system should not return: unsafe advice, leaked secrets, invalid tool arguments, hallucinated citations. When the check fires the harness blocks the turn, substitutes a canned reply, or escalates to a human queue.

The check itself can be many things. A regex or schema validator handles the mechanical cases: a credit card pattern, a JSON payload that fails to parse, a URL against an allowlist. A small classifier such as Llama Guard or OpenAI's moderation endpoint handles category labels like self-harm, hate or weapons. A judge-model call handles the fuzzy cases where policy needs a rubric. Frameworks such as NVIDIA NeMo Guardrails and the OpenAI Agents SDK give the pattern a name and a place in the loop; the mechanism is a wrapper around the model call, not a property of the model itself.

Input vs output guardrails: which does what?

Input guardrails run first and gate whether the model is called at all. Their job is cheap, boring rejection: block obvious prompt-injection payloads, reject requests outside the product's scope, strip PII before the message reaches the provider. They tend to be small and fast, because they run on every turn and their latency stacks on top of every response the user waits for.

Output guardrails run after the model responds and gate what the user sees. Their job is harder because the surface is open ended: the model can produce anything the prompt allowed. Output checks include schema validation for structured output, groundedness checks that compare the answer against retrieved sources, safety classifiers that score the response for category harms, and code checks that reject unsafe shell commands before the harness executes them. When an output guardrail trips, the usual moves are to retry with a stricter system prompt, return a refusal, or route the turn to a human reviewer.

Guardrails vs the system prompt: where does the rule live?

A rule in the system prompt asks the model to enforce policy on itself. It works most of the time and fails exactly when it matters, because a determined user or a poisoned document can override the instruction. A rule in a guardrail is enforced by code that runs regardless of what the model was told, so an injection that flips the system prompt does not flip the check. The trade is latency and cost: every guardrail is another call or another dependency in the response path.

Most teams settle on a working split. Shaping and tone go in the system prompt, hard policy goes in a guardrail. If violating a rule would cause a real incident, such as leaking a customer record, running an unsafe tool, or producing disallowed content, the check has to live outside the model. If the rule is aesthetic, such as answering in Spanish or using bullet points, the system prompt is enough.

Where guardrails fail

The first failure is base rate. A moderation classifier at 99 percent accuracy still flags 1 in 100 clean messages, and on production traffic that means visible refusals on requests the user considered normal. Measure the false-positive rate on your own inputs before you gate a release on a guardrail, not on the classifier's public benchmark.

The second is the coverage gap between what the guardrail scores and what your policy actually forbids. Llama Guard labels a fixed taxonomy of harms; if your policy also forbids financial advice or medical claims, those categories need their own check. Providers keep this explicit: the OpenAI moderation guide lists the categories the classifier covers, and everything else is your responsibility.

The third is the guardrail becoming another attack surface. If the output check is itself an LLM call, a prompt-injection payload in the model's response can target the checker. Keep judge prompts short, do not concatenate untrusted content into the judge's system prompt, and prefer deterministic checks like schemas, regexes and allowlists for anything you would treat as a security boundary.

from agents import (
    Agent,
    GuardrailFunctionOutput,
    InputGuardrailTripwireTriggered,
    Runner,
    input_guardrail,
)
from pydantic import BaseModel

class HomeworkCheck(BaseModel):
    is_math_homework: bool
    reasoning: str

triage = Agent(
    name="Guardrail check",
    instructions="Is the user asking you to do their math homework?",
    output_type=HomeworkCheck,
)

@input_guardrail
async def math_guardrail(ctx, agent, user_input):
    result = await Runner.run(triage, user_input, context=ctx.context)
    return GuardrailFunctionOutput(
        output_info=result.final_output,
        tripwire_triggered=result.final_output.is_math_homework,
    )

support = Agent(
    name="Support",
    instructions="Answer product questions. Refuse anything else.",
    input_guardrails=[math_guardrail],
)

try:
    reply = await Runner.run(support, "solve x^2 - 5x + 6 = 0")
    print(reply.final_output)
except InputGuardrailTripwireTriggered as e:
    # The support agent never saw the message. Log the trip so a
    # dashboard can track false-positive rate against clean traffic.
    print("blocked:", e.guardrail_result.output.output_info.reasoning)