Publications
Open Problems in Mechanistic Interpretability
This review discusses the current frontier of mechanistic interpretability, which aims to understand the computational mechanisms underlying neural networks. While the field has made progress, many open problems remain, including the need for improved methods, better applications to specific goals, and engagement with socio-technical challenges.
Pretraining Language Models with Human Preferences
We find that conditional training of large models (LMs), which learns the distribution over tokens based on human preference scores, reduces undesirable content while maintaining downstream task performance. Pre-training LMs with human feedback leads to better preference satisfaction than traditional LM pre-training followed by feedback-based finetuning.
RL with KL penalties is better viewed as Bayesian inference
We argue that the standard reinforcement learning approach in fine-tuning large language models is flawed and leads to distribution collapse, and propose a Bayesian inference view of KL-regularized RL which explains how it avoids the distribution collapse problem.