Abstract
As their size increases, Large Languages Models (LLMs) are natural candidatesfor network pruning methods: approaches that drop a subset of network weightswhile striving to preserve performance. Existing methods, however, requireeither retraining, which is rarely affordable for billion-scale LLMs, orsolving a weight reconstruction problem reliant on second-order information,which may also be computationally expensive. In this paper, we introduce anovel, straightforward yet effective pruning method, termed Wanda (Pruning byWeights and activations), designed to induce sparsity in pretrained LLMs.Motivated by the recent observation of emergent large magnitude features inLLMs, our approach prunes weights with the smallest magnitudes multiplied bythe corresponding input activations, on a per-output basis. Notably, Wandarequires no retraining or weight update, and the pruned LLM can be used as is.We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2across various language benchmarks. Wanda significantly outperforms theestablished baseline of magnitude pruning and performs competitively againstrecent method involving intensive weight update. Code is available athttps://github.com/locuslab/wanda.