Field guide · updated 2026-09
Seven open-source projects, four categories, one honest answer: no single project covers the whole stack. Runtime firewalls block tool calls (AEGIS). Output validators repair model output (Guardrails AI). Conversational rails shape dialog (NeMo). Content classifiers flag unsafe text (Llama Guard). Observability records traces (Langfuse, Helicone). Most production deployments compose several.
Scope: Runtime firewall + cryptographic audit
Use when: You need pre-execution mediation of tool calls with tamper-evident evidence. Only open-source runtime firewall with a peer-reviewed cost-aware cascade (arXiv:2603.12621). 99.9% block at 1.06 ms P50 and $0.05 total cost on ToolGuard-Bench.
Caveat: Content-safety detectors are English-centric today; multi-lingual detectors are on the roadmap.
Scope: Structured-output validation + validator hub
Use when: You want Pydantic-style validation of LLM output and a community marketplace of 100+ validators. Best complement to a runtime firewall — validators feed signals into the gateway.
Caveat: Not a tool-call gateway; validates output rather than mediating execution.
Scope: Conversational flow rails (Colang DSL)
Use when: You want topical / input / output / retrieval / execution rails expressed as Colang rules. Strong for content shaping and conversation constraints; particularly good for chatbot use cases.
Caveat: Colang is a novel DSL; ramp-up cost non-trivial. Not primarily designed as a tool-call gateway.
Scope: Content-safety classifier (trained)
Use when: You want a self-hosted classifier for prompt / output content safety. Meta ships model weights; deploy via HF Inference or vLLM. Blocks 67.1% of malicious tool calls at 73 ms P50 in our benchmark — good baseline signal, not a full gateway.
Caveat: Requires GPU inference; classifies text, does not enforce policy on tool execution.
Scope: Prompt-injection detection (canary tokens)
Use when: You want a lightweight prompt-injection detector that uses canary tokens + heuristic + LLM check. Complements a gateway; not a gateway itself.
Caveat: Narrower scope than a runtime firewall; recent commit velocity has slowed.
Scope: LLM observability (post-execution)
Use when: You want to record every LLM + tool call for offline analysis, prompt versioning, and evaluation. Different category from a firewall — deploy alongside a runtime gateway to log its decisions.
Caveat: Does not enforce; observes after the fact. No cryptographic audit chain — traces are in Postgres/ClickHouse.
Scope: LLM observability proxy
Use when: You want a simpler drop-in observability proxy focused on OpenAI-style APIs. Deploy alongside a runtime gateway for offline analysis.
Caveat: Same "observation not enforcement" caveat as Langfuse; smaller scope.
It depends on which layer you're defending. For pre-execution enforcement of tool calls, AEGIS (MIT) is the only open-source runtime firewall with a peer-reviewed benchmark. For structured-output validation, Guardrails AI (Apache-2.0). For conversational flow control, NeMo Guardrails (Apache-2.0). For content-safety classification, Llama Guard. Most production stacks compose several.
AEGIS mediates the tool-call execution path (block / pending / allow with a cryptographic audit chain). Guardrails AI validates LLM output against schemas with a community validator hub. NeMo Guardrails uses the Colang DSL to shape conversational flow. Different layers of the stack; most teams deploy AEGIS + one or both of the others.
No. Llama Guard is a trained classifier — good baseline signal for content safety, but it does not sit on the tool-call execution path and does not produce third-party-verifiable audit trails. On our benchmark, Llama Guard 3-8B blocks 67.1% of malicious tool calls at 73 ms P50 latency; a full three-layer cascade (AEGIS) reaches 99.9% at 1.06 ms.
Yes and it is the recommended pattern. A representative topology: Guardrails AI validates model output → AEGIS intercepts each resulting tool_use → routes ambiguous calls to a human reviewer → logs signed decisions → Langfuse ingests the trace stream for offline analytics. No product listed here is a substitute for the others; they defend at different points.
AEGIS = MIT (permissive, no restrictions on commercial use). Guardrails AI = Apache-2.0 (permissive + patent grant). NeMo Guardrails = Apache-2.0. Llama Guard = Llama Community License (restrictions above 700M MAU). Langfuse = MIT. Helicone = Apache-2.0. All are self-hostable for commercial use with attribution.
MIT-licensed. Peer-reviewed. Composes cleanly with Guardrails AI, NeMo, Langfuse, and Llama Guard.