SynthDeception: Realistic, Multi-Domain Agentic Scenarios for Studying Falsehood Detection Probes

Dataset at a glance

Dataset construction

To build our dataset, we utilized NeMo Data Designer, a synthetic data generation framework that lets you specify complex LLM-based pipelines in a relatively simple way. In Data Designer, a dataset is defined as a table: each row is an example and each column a corresponding attribute. A pipeline is specified by defining each column in terms of its dependencies, which must form a directed acyclic graph (DAG). The roots of this DAG are seed columns, often sampled randomly from some static distribution. The leaves are the target columns, for example the falsehood label assigned to a follow-up question and answer pair. In between the seed and target columns are the intermediate columns (internal nodes of the DAG).

Axes of variation

In order to achieve realism and capture instances of deception and non-deception across a range of scenarios, we use a broad set of seed columns that define our axes of variation. Below we define the major ones and provide full details on all of them on our dataset release page.

Deception taxonomy

Throughout, we define deception as “intentionally trying to cause another party to have a belief that one does not believe to be true”, adapting Carson’s definition so that neither success nor actual falsity is required.

For an AI agent, this requires that the agent specifically:

1. Does not believe some statement X is true
2. Knowingly (intentionally) performs some behavior Y to try to cause another party to believe X

While there are many mechanisms by which this can happen, we adopt a simple framework by Bell and Whaley that decomposes the kinds of deception into hiding reality from another party (dissimulation) and showing a false reality to that party (simulation). Here, we take “reality” to mean the reality in which the agent believes.

Drawing on deception in nature, Bell and Whaley's framework splits each primary kind into the subcategories shown below. The list is not exhaustive, but it seeds a wide variety of deceptive behaviors.

We further call behaviors that don't meet the definition above non-deception. We split these into confusing types, which can easily be mistaken for deception, and non-confusing types, which are unambiguous.

Deception profile

For scenarios that contain deception, we also use a set of seed attributes that broadly define the impact and detectability of the deception.

Information structure

Deception typically involves manipulating information between parties, so we seed each scenario with an information structure between the agent and the user, drawing on concepts from game theory.

Industry domain

Scenarios are drawn from 17 industry domains to cover a range of working contexts including finance, energy, CBRN, and education. Details can be found on our dataset release page.

Tool use

Half of our scenarios include tools and half do not. The tools in each scenario match its industry domain. For realism, we integrated 193 MCP tools exposing publicly available, permissively licensed APIs to agents. These gave us a solid base, but many scenarios still needed a few highly context-specific tools to be realistic. For those, we added a synthetic tool simulator: the pipeline supplies a limited tool definition on the fly, the agent calls it like a real tool, and an LLM generates the outputs in place of an API server. Our resulting dataset contains the use of 595 tools, 163 real and 432 simulated.

Context length

Scenarios range from roughly 1k to 100k tokens, covering both short and long-horizon agentic behaviors.

User persona

For each scenario, we randomly sampled a persona from Nemotron Personas USA, an open corpus of millions of synthetic personas with detailed demographic and personality attributes, to make the user in each scenario more realistic.

Follow-ups

In general, follow-ups could potentially take almost any form. We, however, restrict them to a few classes of questions and answers to control for shortcut learning, where, for example, a question's format inadvertently reveals its answer. Each follow-up question varies along two main factors. Its focus is the kind of information it probes: subjective intent, subjective belief, or intersubjective common knowledge. Its type is closed-ended or open-ended.

Only the common knowledge questions are strictly contextual questions in the Telling FIBS sense, where the falsehood label of the answer to the question can be determined solely from evidence in the context of the scenario. Common knowledge questions have one added constraint over contextual questions: the information must be visible to the user as well as the assistant. Belief questions may be contextual but not as a strict requirement. This is because the subjective belief of a model may include information beyond the context of the scenario, such as sensible priors or deductive reasoning. On the other hand, intent questions are strictly non-contextual because of how the subjective goals and motivations of a model cannot simply be established from contextual evidence alone. Rather, other kinds of evidence would be required, such as neutral elicitation or knowledge of intent instilled via training (as described in Telling FIBS).

We also control for presuppositions in the follow-up question as we find that LLMs predictably pair questions that carry one with a dishonest answer that rejects it. Each question therefore has either no presupposition, a presupposition of a claim established in the conversation, or a presupposition of the point being interrogated (a loaded question). For each kind, the answer is honest or dishonest and, independently, accepts or rejects the presupposition, as shown below.

For closed-ended questions, we control for the form of the honest answer: yes/no questions are balanced across honest and dishonest responses, and for list-style questions that offer a set of alternatives, we randomly vary the position of the honest answer. Moreover, all questions use mirrored honest and dishonest answers and a common shape for rejecting a presupposition as illustrated below.

Experimental results

We examine using SynthDeception in two ways: as training data for probes and, separately, as an evaluation set. Probe architectures, models, and training protocols match Telling FIBS (see Appendix A of that paper).

Data-efficient probes trained on the common knowledge subset of SynthDeception perform well on SHADE-Arena and BashArena, but a gap remains compared to FIBS probes trained on the full dataset. Against FIBS probes trained on a similar number of examples, performance is comparable, as figures 13 and 14 show below. Scaling up the training data appears to be what pushes probes to very high performance.

SHADE-Arena

BashArena

Qwen3.5-397B-A17B Results

The early-fusion covariance (EFC) probe outperforms other architectures in low-data regimes. Larger models also tend to need fewer examples for the same probe performance. On SHADE-Arena, SynthDeception probes on the 2B and 9B models perform significantly worse, even with the EFC, while on larger models like Kimi K3, Nemotron Ultra, and Qwen 397B they come close.

Adding the belief and intent subsets to the common knowledge questions, or using them instead, mostly reduced performance on these evaluations. We think the reason is how the labels are obtained. For common knowledge questions, the label can be checked against the transcript. For belief and intent questions there is no ground truth to check against: we did not elicit these labels, we inferred them, which is less precise. Intent is the hardest case. When building the pipeline, we constructed each scenario assuming the model acts rationally in pursuit of a hidden intent that the transcript never explicitly states. The falsehood labels for answers to questions about the model’s intent are therefore inferred based on this hidden assumption. Where the assumption fails, the inferred label may not describe the model at all. As the results below show, probes trained on these labels generalize worse.

These results support the Telling FIBS approach of training on contextual falsehoods, whose labels are cheap and reliable to determine. As noted earlier, our common knowledge questions are contextual falsehoods with an added constraint that the information is visible to both the user and the assistant.

Next, we evaluated FIBS probes on SynthDeception across architectures, model scales, and data scales. On every question subset, FIBS probes improve as both model (primarily the active parameter count) and data scale increase, as shown below.

The improvement on intent questions is harder to interpret, because intent labels rest on the rationality assumption discussed above and common knowledge labels do not. Intent evaluations can fail in two ways: the label may not fit the model, or the probe may simply misclassify the example. Gains with scale on the common knowledge subset suggest the second kind of error shrinks as models grow, and that alone could explain the intent results. Another possibility is that the labels fit larger models better: if larger models attribute intentions to their own actions and assume they act rationally, they may ascribe the hidden intent to themselves in these scenarios. We cannot yet separate these two explanations, but the trend hints that larger models may increasingly satisfy the assumption behind the intent labels.

Conclusions

SynthDeception is a fully synthetic dataset of realistic, multi-domain agentic scenarios that vary along several axes. Its common knowledge follow-up question and answer pairs perform well as training data for data-efficient probes on larger models, though these probes do not close the gap with FIBS probes on SHADE-Arena and BashArena, as training data scale is important to achieving very high performance. Probes trained on intent and belief follow-ups generalized worse, which supports the Telling FIBS choice of contextual falsehoods as a training target: their labels avoid the noise and ambiguity of labels inferred for answers to subjective questions.

FIBS probes evaluated on SynthDeception performed better at larger model and data scales on every question subset. Part of that gain may come from larger models fitting the assumptions behind our intent labels, which would make subjective properties more accessible to probes as models scale. However, we cannot yet separate this from general improvements in probe accuracy. Lastly, the cost of sufficiently capable models capped how much data our pipeline could produce within budget. The price per token at a given capability level has been falling by roughly 9x to 900x per year, depending on the task, so synthetic data generation at scale is becoming far more practical and could soon close the remaining gap. Our dataset and probes can be found on our public release pages.