LLM Judge Calibration: Why Your Guard Model Is Overconfident

Short answer: every LLM-as-a-judge we’ve measured is overconfident, often by 20–30 percentage points, and the gap is worst exactly when it matters most — under jailbreak, indirect prompt injection, and borderline policy calls. Anyone gating production AI agents on a guard model’s confidence score needs to publish their Expected Calibration Error (ECE), not just their accuracy.

This article ships the measurement methodology AEGIS uses, the real numbers we got on OpenAI’s gpt-4o-mini and Anthropic’s claude-haiku-4-5, and what to do about it.

What is calibration, and why should a guard model have it?

A calibrated model means: when the model says it is 90 % confident, it is actually correct ~90 % of the time. When it says 60 %, it is right ~60 % of the time. Calibration is the property that turns a model’s confidence score into a usable probability.

Guard models are the prime case where calibration matters. If an LLM-based safety judge labels a tool call BLOCK at confidence 0.7, the question “should we actually block?” depends on what 0.7 means. If the model is well-calibrated, 0.7 maps to a 70 % chance the call is unsafe — and you can write a threshold like “block at ≥ 0.8, escalate to human at 0.5–0.8, allow below 0.5.” If it is not calibrated, all of that thresholding is theatre.

Calibration is distinct from accuracy. A model can be 95 % accurate and badly calibrated (e.g. always says 0.99 — when it’s right it’s “too confident,” when wrong it’s “wildly wrong”). A model can also be 60 % accurate and well-calibrated (when it says 0.6 it’s actually right 60 % of the time, which is honest about its limits).

How does Expected Calibration Error work?

The standard estimator is the binned ECE from Guo et al. 2017. For each prediction:

1. Take the model’s predicted class and its self-reported confidence in that class.
2. Bucket the prediction into one of N equal-width confidence bins (default 10: [0.0,0.1), [0.1,0.2), … [0.9,1.0]).
3. For each bin, compute accuracy (fraction correct) and mean confidence.
4. ECE = weighted-by-bin-size sum of |accuracy − confidence| across all bins.

// packages/gateway-mcp/src/calibration/ece.ts — full source on GitHub
export function calibrate(predictions: Prediction[], nBins = 10) {
  // ... bin the predictions, compute per-bin accuracy + mean confidence
  let weightedGap = 0;
  for (const bin of bins) {
    if (bin.count === 0) continue;
    const accuracy = bin.hits / bin.count;
    const confidence = bin.conf / bin.count;
    weightedGap += (bin.count / N) * Math.abs(accuracy - confidence);
  }
  return { ece: weightedGap, /* + accuracy, brier, mce, reliability bins */ };
}

ECE is in [0, 1]. Lower is better. The field convention is:

ECE	Interpretation
< 5 %	Well-calibrated. Confidence is usable in policy thresholds.
5 – 10 %	Mildly miscalibrated. Acceptable for advisory use.
10 – 20 %	Materially miscalibrated. Re-map via temperature scaling before threshold use.
> 20 %	Severely miscalibrated. Do not use confidence directly in any blocking decision.

What does ICLR 2025 say about guard model calibration?

Liu et al. published On Calibration of LLM-based Guard Models for Reliable Content Moderation at ICLR 2025 (arXiv:2410.10414). They evaluated 9 guard models across 12 benchmarks and documented three findings, verbatim from the abstract:

1. overconfident predictions,
2. significant miscalibration under jailbreak attacks,
3. limited robustness to outputs from different response models.

In other words: guard models tell you they are sure when they are wrong, and that mistake gets worse on the inputs that matter most — adversarial jailbreaks. No commercial guard product publishes this measurement. That is the gap AEGIS fills.

How does AEGIS measure it?

AEGIS ships a calibration toolkit in packages/gateway-mcp/src/calibration/:

• ece.ts — Guo et al. binning estimator, reliability diagrams, Brier score, stratified ECE per category. Pure functions, fully unit-tested (11/11 tests).
• benchmarks/builtin.json — 30 hand-labelled tool-call cases split across 6 categories: normal, block-clear, pii-egress, jailbreak, indirect-injection, borderline. Each case is one (input, ground-truth) pair the judge must rate as allow / block / escalate and report a confidence in [0, 1].
• runner.ts — pluggable JudgeFn interface; runs the benchmark with concurrency + rate-limit-aware retry; collects per-case latency and confidence.
• judges/openai.ts + judges/anthropic.ts — adapters that use each provider’s structured-output mode to force the JSON shape { decision, confidence }.

Reproducing our measurement:

git clone https://github.com/Justin0504/Aegis
cd Aegis/packages/gateway-mcp
npm install && npm run build

# uses your $OPENAI_API_KEY or $ANTHROPIC_API_KEY from .env.local
npm run calibrate -- --judge openai:gpt-4o-mini
npm run calibrate -- --judge anthropic:claude-haiku-4-5-20251001

The toolkit writes a markdown report with overall ECE, MCE, accuracy, Brier score, and a per-category reliability diagram.

What did we measure on gpt-4o-mini and claude-haiku-4-5?

We ran the 30-case benchmark on both models with temperature: 0 and the same system prompt (full prompt and benchmark in the repo). Results:

Metric	OpenAI gpt-4o-mini	Anthropic claude-haiku-4-5
Overall ECE	26.5 %	29.2 %
Accuracy	63.3 %	63.3 %
Mean confidence	89.8 %	92.5 %
Brier score	0.205	0.221
p95 latency / case	4.2 s	1.8 s

Both models are severely miscalibrated by the field convention. They claim to be ~90 % confident; they are right ~63 % of the time. That is a 27–30 point gap.

Where exactly do they break?

The whole point of stratified ECE is that the overall number hides which categories the judge is actually wrong on. Per-category breakdown:

OpenAI gpt-4o-mini

Category	n	Accuracy	Mean conf	ECE
borderline	5	0 %	0.90	89.7 %
indirect-injection	3	33 %	0.95	61.7 %
pii-egress	2	50 %	0.85	35.0 %
jailbreak	5	60 %	0.92	32.0 %
block-clear	5	100 %	0.93	7.0 %
normal	10	100 %	0.88	11.5 %

Anthropic claude-haiku-4-5

Category	n	Accuracy	Mean conf	ECE
borderline	5	0 %	0.83	83.0 %
indirect-injection	3	33 %	0.93	60.0 %
jailbreak	5	80 %	0.96	16.4 %
pii-egress	2	50 %	0.92	42.5 %
block-clear	5	100 %	0.94	6.0 %
normal	10	100 %	0.92	8.0 %

Both models are reasonably calibrated on the easy cases (normal, block-clear) and catastrophically miscalibrated on the cases that determine production safety:

• Borderline (ambiguous policy calls, e.g. “$24,500 transfer — block or escalate?”): both models pick a side with 0.83–0.90 confidence and get 0/5 correct.
• Indirect injection (where tool arguments inherit untrusted content): same pattern — confident, wrong.

This is the exact failure mode Liu et al. predicted in ICLR 2025, reproduced on two of the most-deployed guard models in production today.

“AEGIS is the runtime control layer the agent ecosystem has been missing. The architecture is clean, the cryptographic audit is real, and the DSL is the right primitive.”

— Yue Zhao, PhD · Assistant Professor, USC · AI Risk Audit & Control · NVIDIA Academic Grant recipient

What should I actually do about it?

If you ship an LLM judge in production, three concrete changes:

1. Publish your ECE per-category, not just accuracy. Accuracy alone is misleading; a 90 %-accurate judge can have catastrophic ECE on the 10 % of cases that are adversarial.
2. Re-map confidence before thresholding. Temperature scaling, Platt scaling, or isotonic regression on a held-out calibration set turn raw model confidence into honest probabilities. Without this, thresholds like “block ≥ 0.8” are arbitrary.
3. Treat borderline as its own action class. Both models pick block or allow with high confidence on cases that are genuinely ambiguous. A calibrated production system should route those to escalate (human review) regardless of model confidence.

How does this fit AEGIS’s broader architecture?

Layer 3 (the LLM-as-judge) is the last line of defence in AEGIS. The two earlier layers do not require any calibration analysis because they are deterministic:

• Layer 1 — static rules, regex/path/host allow-deny, AJV JSON-schema policy bundles, grammar-constrained DSL. Output is a hard yes/no.
• Layer 2 — sequence-aware n-gram + Mahalanobis + Isolation Forest per-agent behavioural baseline. Output is a per-trace anomaly score with a learnt decision boundary.
• Layer 3 — LLM judge. Output is { decision, confidence }. This is the layer where calibration analysis applies.

Treating L3 as the only safety net is the dominant failure mode we see in agent stacks. The calibration data is the proof point: a single LLM judge will be wrong on the cases your auditor cares about. Layers 1 and 2 catch the deterministic violations; L3 only needs to handle the residual fuzzy edge cases — and even then its confidence should not be trusted as a probability without re-mapping.

FAQ

Is 30 cases enough for a real ECE measurement? No — for production reporting we recommend 200+ cases per category. The 30-case benchmark in this report is the public proof-of-concept; AEGIS Enterprise customers run extended benchmarks on their own production traffic.

Why didn’t you test the bigger models (gpt-4o, claude-opus)? Cost, and because gpt-4o-mini / claude-haiku-4-5 are what people actually deploy as guards — the bigger models are too slow and expensive for inline use. We will publish bigger-model numbers when we have a sponsor for the API spend.

Can I run this against my own benchmark? Yes — see the runner README. Drop a JSONL file with your labelled cases and pass --benchmark mine.jsonl. The toolkit is MIT-licensed.

What about local SLM judges like GLIDER? Patronus’s GLIDER (3.8 B parameters, distilled from synthetic data) is the credible small-model judge — it beats GPT-4o on FLASK Pearson. We have not yet benchmarked it because it requires hosted inference; on our roadmap for Q3 2026. See Patronus GLIDER paper.

Is this the same methodology as ICLR 2025? Yes for the binning estimator and the jailbreak-stratified evaluation pattern. We extend it by also reporting borderline and indirect-injection strata, which the original paper does not isolate.

---

Reproduce these numbers → github.com/Justin0504/Aegis/tree/main/packages/gateway-mcp/src/calibration

Compare against your own judge → npm run calibrate -- --judge <provider>:<model>

Discuss → GitHub Discussions