Muown Achieves Breakthrough in Angular Step-size Decay for Transformer Optimization

Recent advancements in matrix-aware optimizers, particularly Muown, have demonstrated significant performance improvements for pre-training Transformers. Muown utilizes a unique approach by separating weight matrices into row magnitudes and an un-normalized direction variable, with the former updated using Adam and the latter through Muon. This research establishes that the directional update in Muown mirrors a Riemannian step on normalized directions, while the magnitude modulates angular step size, enhancing step-size stability. The new method, AngularMuown, optimizes over normalized directions with a decoupled schedulable angular multiplier, outperforming Muown and currently leading in the modded nanoGPT speedrunning competition. Further tests on Qwen2-0.5B and 1.1B parameter mixture-of-experts models indicate the algorithm's scalability. The implementation is accessible at https://github.com/fhueb/angular-muown.