Introducing GiVA: A Gradient-Based Strategy for Efficient Vector-Based Adaptation
GiVA, a novel gradient-based initialization strategy for vector-based adaptation, offers a parameter-efficient alternative to traditional fine-tuning methods like LoRA. As model sizes increase, fine-tuning techniques have needed to adapt, with vector-based methods gaining traction due to their efficiency. However, they often require higher ranks than LoRA, leading to increased training costs. GiVA addresses this issue by achieving training times comparable to LoRA while maintaining the extreme parameter efficiency characteristic of vector-based adaptation. Evaluated across various benchmarks, including natural language understanding, generation, and image classification, experiments indicate that GiVA consistently outperforms or matches the performance of existing methods, significantly reducing rank requirements by a factor of eight.
arXiv AINeeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky et al.Read →