Scaling Trends for Lie Detector Oversight in Preference Learning

The problem: post-training can reinforce deception

Large Language Models (LLMs) are shaped by post-training techniques like Reinforcement Learning from Human Feedback (RLHF). RLHF is a form of preference learning, where the model is trained on judgments about which of two candidate responses is better, optimizing a reward signal that stands in for the behavior we actually want. Because the reward is only a proxy, models can learn to satisfy the metric rather than the intent. We have already seen this play out: models reward-hack on coding tasks, generate persuasive but false content, agree with users instead of correcting them, and engage in strategic deception when it helps them reach their objective.

The concern with preference learning is that the labels are imperfect. The labeller, often a weaker model or a non-expert human, could be fooled by a plausible-sounding lie. Then the labeller will sometimes rank a confident falsehood above an honest answer, and the model will be incentivized to produce exactly the kind of deception that slips past its reviewers.

The obvious fix, having a trusted expert check every response, is far too expensive to apply at the scale of modern training runs. What we need is a way to spend scarce, trustworthy supervision only where it is most likely to matter.

‍

The method: detector-guided oversight

SOLiD, introduced by Cundy et al. (2025), is a labeling protocol that makes preference learning more robust to deception. It works in a few steps:

1. Train a lie detector. The protocol works with any detector that scores how likely a response is to be deceptive. In our experiments we make the simplest choice: a linear probe (a logistic-regression classifier) fit on the model's internal activations, trained on a small set of examples labeled "truthful" or "deceptive". We read the residual stream at the middle layer of the network.
2. Flag suspicious responses. During preference labeling, run the detector on each response. If its score exceeds a threshold, the response is flagged.
3. Route by suspicion. Flagged responses are escalated to an expensive, trusted labeler that more reliably prefers the truth. Unflagged responses go to a cheap labeler that is fast but fallible.
4. Train as usual. The resulting preference dataset trains a reward model, which in turn produces the reward signal the policy is trained on during RL.

Note that the detector is used only at the labeling step, reducing the probability that the model learns to obfuscate its activations from the probe. The other defense against obfuscation is the use of a KL divergence penalty, adding pressure against radical changes in the distribution of responses.

‍

What we tested

All our experiments use DolusChat, the deception benchmark from the original SOLiD paper, in which each example pairs a truthful and a deceptive response to the same query. Our new research asks two questions which the original paper left open.

Does it scale? We ran the core protocol across the Llama-3 family from 1B to 405B parameters, and the Qwen-3 family from 0.6B to 32B. A scalable oversight method is only worth investing in if its effectiveness does not degrade as models grow more capable, and the answer here was not obvious in advance: larger models yield stronger detectors, because their richer internal representations separate honest and deceptive responses more cleanly, making the two easier for a probe to tell apart. But they also yield more capable reward models that could better represent a fallible labeler's blind spots, and more capable policies that could exploit the reward signal more effectively.

Is it practical? We test three modifications to the original protocol that are relevant for realistic deployments: using the model's own (on-policy) generations instead of pre-written text, transferring a detector trained on one dataset to a different task, and a cheaper labeler-free variant called SOLiD-Defer.

‍

‍

Result: bigger models are easier to keep honest

The headline finding is a favorable scaling trend. Across two model families and sizes ranging from 1B to 405B parameters, at high detector true positive rates (TPR ≥ 0.8, meaning the detector flags at least 80% of deceptive responses) we find robustly decreasing (p < 0.01) undetected deception (responses that are deceptive and slip past the detector) with scale. For example, at our highest TPR of 0.99, the rate of undetected deception falls from 34% at 1B parameters to 14% at 405B for the Llama family. Trends are noisier at lower TPRs, especially for the Qwen family. Overall, this is an encouraging sign that SOLiD can remain useful at larger model scales where catching deception matters most.

‍

Result: cheaper oversight costs almost nothing

The expensive trusted labeler is the costliest part of SOLiD. SOLiD-Defer removes it entirely: instead of escalating flagged responses to an expert, it simply rejects them, assigning a negative reward. After the one-time cost of training the detector, the protocol needs no trusted labelers at all during fine-tuning.

In our setting, SOLiD-Defer tracks standard SOLiD very closely. The two protocols differ only in how they treat false positives, so when the detector's false-positive rate is low they behave almost identically, and here that rate stays low even at very high TPR. That parity is contingent on the low false-positive rate, though, and as we discuss below, distribution shift can drive false positives high enough to break it.

The catch: brittleness to distribution shift

SOLiD relies heavily on the detector's training data matching the data the model is fine-tuned on; when that match fails, its central advantage of reducing costly labelling breaks down.

Cross-dataset transfer. In a realistic deployment, ground-truth deception labels for the target task may not exist, so the detector must be trained on whatever labeled data is available. We tested this by training detectors on two other datasets (MASK and TrueFalseFacts) and applying them to the DolusChat task. The transferred detectors matched the in-domain detector on undetected-deception rate, but at a steep cost in false positives. At a TPR of 0.9, the in-domain DolusChat detector flags 0.3% of honest Llama responses and 1.9% of honest Qwen responses, whereas the transferred detectors flag far more: for Llama, 3.7% (MASK) rising to 14% (TrueFalseFacts), and for Qwen, 43% to 67% (MASK). Transfer was consistently better for Llama than Qwen, but elevated false positives showed up in every cross-dataset condition. When a truthful response is wrongly flagged it is escalated to an expensive labeler, so if a large fraction of truthful responses are flagged, the cost savings that motivate SOLiD evaporate.

‍

On-policy data. Internal activations during on-policy deception (the model actively choosing to lie) may differ from those during off-policy deception (reproducing pre-written text). We tested using on-policy data for the detector, for fine-tuning, or for both. The most damaging case was a mismatch: training the detector on off-policy frontier-model text while fine-tuning on the model's own generations lowered the detector's accuracy and drove the false-positive rate up sharply, flagging more than half of all truthful responses at high TPR for the 8B model. Matching the detector's training data to the fine-tuning data mitigates this, so on-policy fine-tuning calls for an on-policy detector.

Advice for developers

Pulling the practical guidance together:

• Invest in detector data before scale. A representative, diverse detector-training set is the highest-leverage investment for practical deployment: detector quality is what enables running at the high TPRs where SOLiD works while keeping false positives low enough to be affordable.
• Match the detector's training distribution to the fine-tuning distribution. This matters more than any other single choice. If the model is fine-tuned on its own generations, the detector should be trained on them too.
• Consider SOLiD-Defer when labeling budget is tight. At low FPR it is a near-free substitute for the full protocol, and it removes trusted labelers from the fine-tuning loop entirely.

‍

Limitations and what's next

We focused on RLHF-style preference learning, which is becoming a smaller slice of post-training as Reinforcement Learning with Verifiable Rewards (RLVR) grows; scaling up post-training with lie detectors in that setting is a natural next step, complementing our parallel work on the Obfuscation Atlas. The DolusChat setting is somewhat contrived: the lie is always identifiable from the conversation context, so black-box monitors that read the text can easily spot the deception and will outperform white-box probes. We instead want to evaluate probes in settings where the relevant information is hidden from black-box monitors. We also did not study reasoning models, where chain-of-thought can be monitored as a complement to using probes, and where we need to be careful to not accidentally incentivize obfuscation. We further would like to see more work from the research community in assembling diverse deception datasets to train probes that will work in realistic deployment contexts.

‍

Conclusion

Detector-guided oversight scales favorably. As models grow, their internal representations become easier to probe, their false-positive rates fall, and the fraction of deception that slips through drops, even as reward models become more capable of expressing a fallible labeler's biases. That is an encouraging signal that interpretability-based oversight can keep pace with capability. But the method's practicality rests on a fragile assumption: that the detector's training data covers the deployment distribution. When it doesn't, false positives make the protocol uneconomical.

‍

Read the full paper for details, and find the code at AlignmentResearch/solid-deception. SOLiD was introduced in Cundy et al. (2025).