Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

The Problem: When Syntax Masks Semantics

Which training examples teach models to generate insecure code or exhibit harmful behaviors? Traditional TDA methods, like standard influence functions, are susceptible to ``surface-level’’ correlations. By relying on single test examples, they often prioritize lexical overlap – essentially finding training data that looks like the query output rather than data that actually causes the target behavior of interest. Simple keyword search can rival sophisticated influence methods on fact retrieval tasks because the most influential examples often just share words with the test input, not meaning. This syntax-level bias makes them a poor fit for diagnosing abstract safety risks like sycophancy or emergent misalignment, which aren't tied to any one specific set of words. To fix this, we shift the perspective: instead of querying a single prompt, Concept Influence attributes model behavior to semantic directions using interpretable vectors like linear probes or Sparse Autoencoder (SAE) features.

Method: Concept Influence

At its core, Concept Influence generalizes the standard influence function formulation by replacing the gradient of a single test output with the gradient of a semantic concept vector. By using interpretable directions like linear probes, Sparse Autoencoder (SAE) features, or even crosscoder features – which we show in the full paper can unsupervisedly detect data inserting "sleeper agent" triggers without any prior knowledge of the backdoor – we can query the model about behaviors rather than specific strings. 

A key theoretical insight of our work is providing a bridge between theoretically rigorous attribution and computational efficiency in practice: we show that common, efficient techniques, like Projection Difference (a method from Anthropic’s Persona Vectors) and simple linear probes for finding maximally activating data (Vector Filter), are actually first-order approximations of the Concept Influence formulation. These approximations run over 20× faster than gradient-based methods while maintaining competitive performance when training and evaluation domains align closely.

Experiment: Preventing Emergent Misalignment

To demonstrate the effectiveness of Concept Influence and efficient approximations compared to standard influence functions, we first evaluated them in the context of Emergent Misalignment (EM), where narrow fine-tuning on one domain causes broad, undesirable traits – like "evil" or sycophantic behavior – to appear in the fine-tuned model. We focused on attributing the "evil" trait by fine-tuning Qwen2.5-7B on datasets containing misaligned opinions, bad medical advice, insecure code, and math errors.

Across all domains, Concept Influence consistently outperformed traditional influence functions, providing a cleaner separation between aligned and misaligned training examples. Strikingly, we found that training on just the top 10–20% of "influential" data identified by several could produce models significantly more "evil" than those trained on the full dataset. For insecure code, the full dataset yielded models with roughly 1% "evil" scores, while the influence-selected subset produced scores around 10%.

We also confirmed that these results are robust across different architectures and behaviors; the results for Sycophancy and evaluations on Llama3.1-8B can be found in our appendix, where the performance advantages of concept-based attribution remain broadly consistent.

To understand what exactly makes a data point influential, we moved beyond just looking at individual examples and leveraged Sparse Autoencoders (SAEs) to group training data into semantically coherent clusters. This approach reveals a stark qualitative difference: while traditional influence functions often get distracted by "distractor" tokens in a prompt – highlighting generic clusters like legal frameworks or tax terms simply because they appeared in the text – Concept Influence hones in on features semantically similar to the behaviour. For instance, when analyzing a query about violent revenge, our method surfaced clusters related to historical oppression, criminality, and societal critique. 

‍Experiment: Post-Training on OASST1

While emergent misalignment benchmarks provide a controlled environment, we also tested these methods on a non-synthetic, real-world post-training dataset: OpenAssistant Conversations (OASST1). Our goal was to maximize instruction-following capability while minimizing harmful "evil" behavior using as little data as possible. We observed a clear trade-off: as training data increases, the model's instruction-following (MTBench score) improves, but its safety (Evil score) deteriorates.

Filtering the dataset with Concept Influence and its efficient approximations significantly improved this Pareto frontier. Notably, the simple and efficient Vector Filter method achieved the strongest results—using only 5% of the "safest" data, we were able to match the capability of a model trained on 100% of the dataset while keeping the evil score substantially lower. These results demonstrate that interpretability-based attribution can be used as a practical tool for data curation that makes LLMs safer without sacrificing performance.

‍When to Use What

Our analysis revealed two distinct families of attribution methods with different strengths:

Gradient-based methods (including Concept Influence with full gradients) perform better under severe distribution shift, where fine-tuning and evaluation domains differ substantially. They're more robust but computationally expensive.

Probe-based methods (Vector Filter, Projection Difference) work well when domains align closely and run over 20x faster. Because they operate entirely on the base model, they let practitioners anticipate data effects without fine-tuning.

Conclusion

Ultimately, understanding why a model behaves the way it does starts with understanding the data that built it. If we can do this, we move one step closer to making LLMs not just more powerful, but more predictable and safe during the training and fine-tuning stages. Specifically, by shifting the focus of training data attribution from specific strings to abstract semantic directions, we can build a more robust and scalable toolkit for AI safety. 

Read the full paper for more information.