Abstract
We present a novel Parameter-Efficient Fine-Tuning (PEFT) method, dubbed asAdaptive Freezing of Low Rank Adaptation (AFLoRA). Specifically, for eachpre-trained frozen weight tensor, we add a parallel path of trainable low-rankmatrices, namely a down-projection and an up-projection matrix, each of whichis followed by a feature transformation vector. Based on a novel freezingscore, we the incrementally freeze these projection matrices during fine-tuningto reduce the computation and alleviate over-fitting. Our experimental resultsdemonstrate that we can achieve state-of-the-art performance with an averageimprovement of up to $0.85\%$ as evaluated on GLUE benchmark while yeilding upto $9.5\times$ fewer average trainable parameters. While compared in terms ofruntime, AFLoRA can yield up to $1.86\times$ improvement as opposed to similarPEFT alternatives. Besides the practical utility of our approach, we provideinsights on the trainability requirements of LoRA paths at different modulesand the freezing schedule for the different projection matrices. Code will bereleased.