Tomek Korbak

We present 11 instances of inverse scaling: tasks where language models get worse with scale rather than better, selected from 99 submissions in an open competition, the Inverse Scaling Prize.

We introduce Imitation Learning from Language Feedback (ILF), demonstrate that large language models accurately incorporate natural language feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries.

We introduce Imitation Learning from Language Feedback (ILF) to improve code generation, demonstrating that a small amount of natural language feedback during training can lead to significant performance gains on program synthesis benchmarks.

We find that conditional training of large models (LMs), which learns the distribution over tokens based on human preference scores, reduces undesirable content while maintaining downstream task performance. Pre-training LMs with human feedback leads to better preference satisfaction than traditional LM pre-training followed by feedback-based finetuning.

We argue that the standard reinforcement learning approach in fine-tuning large language models is flawed and leads to distribution collapse, and propose a Bayesian inference view of KL-regularized RL which explains how it avoids the distribution collapse problem.