SafeSteer Introduces Localized On-Policy Distillation for Enhanced Safety Alignment in LLMs

Researchers have proposed a novel method called SafeSteer to improve the safety alignment of Large Language Models (LLMs) without compromising their general capabilities. The study highlights the challenges associated with the alignment tax, which occurs when aligning LLMs with human values. SafeSteer employs on-policy distillation focused on safety tokens, using a safety teacher created through activation steering. The method includes a safety token selection algorithm that minimizes the reverse KL penalty during training. Experimental results demonstrate that SafeSteer outperforms existing methods, achieving strong safety performance on seven benchmarks with minimal degradation on five general capability benchmarks, while requiring significantly fewer harmful samples for alignment. More details are available on the project's webpage.