Zhijing Jin

We built TamperBench, a unified framework for evaluating the tamper resistance of open-weight LLMs, addressing the lack of standardized benchmarks in this area. It evaluates 21 models across nine attack types with systematic hyperparameter sweeps, covering both safety and utility metrics. Key findings include that jailbreak-tuning is generally the most severe attack and that Triplet is the strongest alignment-stage defense.

As large language models gain popularity, their vulnerability to adversarial attacks remains a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Misalignment, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity within our experimental datasets. We then evaluate the adversarial performance of these fine-tuned models and assess how dataset factors correlate with attack success rates. Lastly, we explore potential causal links, offering new insights into adversarial defense strategies and highlighting the crucial role of dataset design in preserving model alignment.