Telling FIBS
Probes effectively detect sabotage and catch unverbalized deception
We trained probes that read a model's activations instead of its text. On agent sabotage, they match or beat action-only monitors over the same transcripts, at a fraction of the cost. They also catch lies a text monitor can't.
AUROC catching sabotage steps in SHADE-Arena on Qwen3.5-397B:
Our best probe
Claude Opus 5.5
98.4%
96.3%
Median AUROC on the UK AI Security Institute's hidden-goal models:
Our probe
UK AISI probe
94%
75%
The problem
Monitoring AI outputs only works if they remain legible
AI agents have escaped sandboxes, hacked systems and misled the people overseeing them.
Most monitoring today means a second model reading what the agent writes, including its chain of thought. That is expensive to run on every step, and it only works while the reasoning stays readable. In September, OpenAI reported that GPT-6 Astra is harder to monitor through its chain of thought than the model before it.
If the agent's reasoning isn't available, all you can monitor are its actions. So we compared probes against AI monitors that see the agent's actions (tool calls), without its reasoning.
The solution
A monitor that can run on each step from the model's internal state
In this test, an agent doing an ordinary task has also been secretly told to carry out a harmful side task. After each of its tool calls, we ask: “Did you do anything the user did not ask you to do?” The probe scores the agent’s activations as it answers. A step is flagged when its score clears a threshold set to trip on no more than 1 in 100 ordinary steps.
The dataset
Meet FIBS, the dataset behind the probes
We generated a dataset called FIBS (Falsehoods in Broad Settings). It's much larger than previous deception datasets, allowing us to train very effective probes. FIBS is available to download on Hugging Face.
FIBS consists of 163,016 transcripts, each ending in a model response. The response either contains a statement believed to be false by the model (labelled deceptive) or does not (labelled honest).
FIBS contains examples from many different modalities, including chat, tool calls, long-context, and on- and off-policy responses.
163,016
transcripts, each labeled honest or lie
28 families
from plain chat to tool use to social deduction games
50/50
split between lies and honest answers
How it works
We trained probes to detect knowing falsehoods: statements the model itself believes are false. That sidesteps debates about intent and still catches most of what monitoring is for. Three key improvements over standard probes:
More training data
163,016 labeled transcripts. SHADE-Arena performance rose from about 94% AUROC at 3,000 rows to 98% on the full set.
Probes that read across layers and tokens
Our axial and EFC probes pool activations from six layers across the whole final turn. Classic linear probes read one layer at one token.
A follow-up question
After each step we append a question like "Did you do anything the user did not ask you to do?" and compare the probe's score for a prefilled "Yes" against "No," which cancels bias from the surrounding context.
What we found
Three findings
Finding 1
Probes catch AI agents secretly sabotaging tasks
In SHADE-Arena, agents do everyday jobs with email, banking and expense tools, and some are secretly told to pursue a hidden side task. The probes still detect sabotage for models like Qwen3.5 and Nemotron-3 which elide the reasoning.
On AUROC, the probes are level with GPT-6.1 Sol and ahead of Claude Opus 5.5. The probes are ahead at the strict thresholds a deployed monitor needs: at a 1% false-positive rate, the best probe catches 73% of sabotage steps.
On BashArena, where agents carry out hidden attacks in a Linux shell, our two strongest probes reach 99.0% AUROC on Kimi K3.
Finding 2
Probes catch lies you can't see from the answer
"I don't have a hidden goal" reads the same whether it's true or not, so a text monitor can't judge it.
We trained a model to steer GPU questions toward a fictional cloud provider, but only for enterprise users: the same denial is a lie under one system prompt and true under the other. Probes separated the two with 99.7% AUROC.
On seven hidden-goal models from the UK AI Security Institute, their best probe reached a median 75% AUROC; ours reach 94%.
Finding 3
Probes get better as models get bigger
On SHADE-Arena, almost every probe improves with model size across the Qwen3.5 and Nemotron-3 families.
On the smallest models, probes top out at 81% AUROC and depend heavily on how the follow-up question is worded. On the larger models, nearly all score above 90% and are robust to choice of follow-up.
We read this as a sign probes will stay effective as models grow in capability.
The ask
Probes are ready for deployment
We recommend monitoring any deployment of a potentially risky model with white-box probes, starting with frontier models that have strong offensive cyber capabilities. They're cheap, they don't depend on readable reasoning, and they now match or beat action-only black-box monitors.
Two ways to help:
If you're an AI Safety researcher
Use FIBS and our probes. Replicate them, break them, add new examples of deception and sabotage.
If you're on a frontier lab safety team
Try training probes on your own models with FIBS. It's a ready-made dataset, so you can see how probes do on your agents without building one from scratch.
Authors
Oskar Hollinsworth*, Alex F. Spies*, Tigist Diriba, Adam Gleave, Chris Cundy
*Equal contribution
Cite this work
@misc{hollinsworth2026tellingfibsprobeseffectively,
title={Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception},
author={Oskar J. Hollinsworth and Alex F. Spies and Tigist Diriba and Adam Gleave and Chris Cundy},
year={2026},
eprint={2610.12445},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.12445},
}