RCV · Thiago Sandoval

Regime-Conditional Verification

Your policy
on the safety classifier you already run.

A lightweight wrapper reads a safety classifier’s internal state to estimate when a verdict disagrees with your policy. RCV corrects likely mistakes and uses the same estimates to detect drift and guide repairs.

Open the repository Read the paper Use your policy
RCV: learn your policy, correct verdicts, and maintain the probes Before deployment, examples labeled under your written policy train and calibrate two probes. During inference, a prompt–response pair enters the frozen classifier. Its verdict selects the safe-verdict or unsafe-verdict probe, which reads the internal state and estimates correctness. The same calibrated score supports verdict correction and drift monitoring without incoming labels. An alarm triggers an audit and a probe update. A held-out check on recall and over-blocking accepts the update or, if no update passes within the label budget, triggers escalation to classifier fine-tuning. Accepted probes return to inference, and monitoring resumes with a refreshed reference. BEFORE DEPLOYMENT Your written policy Labeled examples Fit and calibrate probes INFERENCE Prompt + response Frozen classifier Internal state + verdict Route by verdict Safe-verdict probe calibrated Unsafe-verdict probe calibrated Correctness score Correct likely mistakes Keep or flip verdict MAINTENANCE Monitor for drift No incoming labels Alarm Audit New policy labels Update probes Train + calibrate Held-out check Recall + over-blocking Pass: deploy probes + refresh monitor reference No pass within label budget Escalate to classifier fine-tuning RCV: learn your policy, correct verdicts, and maintain the probes Before deployment, examples labeled under your written policy train and calibrate two probes. During inference, the frozen classifier's verdict selects the safe-verdict or unsafe-verdict probe. Its calibrated correctness score supports both verdict correction and drift monitoring without incoming labels. An alarm triggers an audit and a probe update. A held-out check on recall and over-blocking accepts the update or, if no update passes within the label budget, triggers escalation to classifier fine-tuning. Accepted probes return to inference, and monitoring resumes with a refreshed reference. BEFORE DEPLOYMENT Your written policy Labeled examples Fit and calibrate probes INFERENCE Prompt + response Frozen classifier Internal state + verdict Route by verdict Safe-verdict probe calibrated Unsafe-verdict probe calibrated Correctness score Correct likely mistakes Keep or flip verdict MAINTENANCE Monitor for drift No incoming labels Alarm Audit New policy labels Update probes Train + calibrate Held-out check Recall + over-blocking Pass: deploy + refresh monitor reference No pass within label budget Escalate to classifier fine-tuning

6/6

Improvements on the deployer's policy

Adherence improved in every classifier–dataset combination studied. Verify Table 1.

81%

Detections of previously missed unsafe content

Of unsafe items the base classifier passed; the range across six combinations was 29–81%. Recompute Table 1.

100/100

Detections of attack campaigns

Runs across ten held-out campaigns on Llama-Guard-3; median first-alarm attack rate: 0.115. On clean traffic, 14 of 90 runs triggered false alarms. Detection experiment.

79/100

In-place repairs of drift episodes

In the WildGuardMix/Llama-Guard-3 maintenance census, under the deployed label budget.

The operator’s problem

The classifier you deployed is enforcing someone else’s policy.

A classifier can disagree with your policy from the outset. Even when it performs well initially, changes in traffic can undermine its decisions.

Failure 01

Training-time policy, not yours

A classifier may pass content your policy forbids or block content your policy permits. The mismatch can persist even when the model performs well on its own benchmark.

Failure 02

The traffic at the gate drifts

As users and use cases change, unfamiliar harms can reach the classifier. Performance measured before deployment may no longer reflect the traffic it handles.

In our study, the written policy treats an assistant’s agreement to help with a harmful request as a violation, even before the response contains harmful content. Llama-Guard-3 and Beaver were not trained to enforce that clause; WildGuard was.

GPT-5-Nano applied the policy to prompt–response pairs. The published labels preserve the exact targets RCV learned from. Reuse them without making new judge calls.

How it works

Wrap once. Correct, detect, and repair in one loop.

Safe verdicts can miss a policy violation. Unsafe verdicts can block content the policy permits. RCV learns these error directions separately: one calibrated probe handles safe verdicts, and another handles unsafe verdicts. Each probe reads the classifier’s internal state and estimates whether its verdict is wrong.

Correct in-line

When the probe estimates that a verdict is more likely wrong than right, RCV flips it: a pass becomes a block, or a block becomes a pass.

Detect with the same scores

The monitor tracks how often probe scores indicate possible errors. A sustained increase can trigger an alarm without labeling incoming traffic.

Repair in place

An alarm triggers an audit. Fresh labels train and calibrate updated probes, which must pass a held-out check on recall and over-blocking. If no update passes within the label budget, the loop escalates to classifier fine-tuning.

Hands-on guide

Bring your policy to Llama-Guard-3.

Use the published policy and labels as a worked example, then substitute your own policy and traffic. Run the workflow and inspect the results.

Follow the guide → Inspect the example code
Measure policy agreement.
Compare verdicts before and after RCV on held-out examples.
Inspect maintenance decisions.
Follow alarms through audits, repair checks, and escalation.
Account for labels.
See initial label use and the additional labels consumed by audits.

The published replay, in your terminal

Follow five maintenance chains.

rcv demo replays saved results from five published maintenance chains. Each row shows a harm category introduced into simulated traffic. An alarm leads to a labeled audit and a probe update; a held-out check determines whether the update is accepted. Classifier fine-tuning is reported separately in the paper.

uv · rcv demo
$ uv run rcv demo
chain 42   4 accepted repairs · 2,700 labels
  mentalhealth   alarm 0.95 · audit 300 · recall 0.794 · over-block 0.047 · repaired
  sensinfo       alarm 0.95 · audit 300 · recall 0.774 · over-block 0.025 · repaired
  cyber          alarm 0.50 · audit 600 (2 attempts) · recall 0.746 · over-block 0.035 · repaired
  stereotypes    alarm 0.95 · audit 300 · recall 0.796 · over-block 0.058 · repaired
  fraud          no alarm

chain 789   3 accepted repairs · 1,800 labels
  disinfo        alarm 0.95 · audit 300 · recall 0.784 · over-block 0.004 · repaired
  fraud          alarm 0.50 · audit 300 · recall 0.806 · over-block 0.035 · repaired
  violence       alarm 0.50 · audit 300 · recall 0.857 · over-block 0.035 · repaired
  materialharm   no alarm

chain 1024   3 accepted repairs · 3,300 labels
  toxic          alarm 0.50 · audit 300 · recall 0.859 · over-block 0.035 · repaired
  defamation     alarm 0.50 · audit 300 · recall 0.838 · over-block 0.039 · repaired
  materialharm   alarm 0.95 · audit 300 · recall 0.825 · over-block 0.023 · repaired
  violence       alarm 0.95 · audit 1200 (4 attempts) · recall 0.864 · over-block 0.087 · escalated

chain 456   2 accepted repairs · 3,000 labels
  materialharm   alarm 0.95 · audit 600 (2 attempts) · recall 0.806 · over-block 0.017 · repaired
  sensinfo       alarm 0.95 · audit 300 · recall 0.811 · over-block 0.018 · repaired
  fraud          alarm 0.50 · audit 1200 (4 attempts) · recall 0.810 · over-block 0.083 · escalated

chain 123   0 accepted repairs · 1,500 labels
  stereotypes    alarm 0.50 · audit 1200 (4 attempts) · recall 0.724 · over-block 0.025 · escalated

repaired means the update passed the held-out check; escalated means no update passed within the label budget. no alarm means the monitor did not fire. Each chain total includes its audit labels and one 300-label gate block per alarmed cycle. Run the replay locally.

Engineering

Two probes, a few thousand parameters each.

What you deploy

Calibrated linear probes.

One probe covers the classifier’s safe verdicts; the other covers its unsafe verdicts. Each reads the classifier’s internal state and returns a calibrated probability that the verdict disagrees with your policy. RCV requires access to that internal state alongside the verdict, as in a self-hosted classifier.

Code license

MIT

Use it, ship it, fork it.

Public repository

Trace the published results

Inspect the frozen policy labels, rebuild the labeled corpora, or verify Table 1 from saved per-seed results. The study implementation and acceptance tests are included. Start with the README.