Abstract
Recent foundational language models have shown state-of-the-art performancein many NLP tasks in zero- and few-shot settings. An advantage of these modelsover more standard approaches based on fine-tuning is the ability to understandinstructions written in natural language (prompts), which helps them generalisebetter to different tasks and domains without the need for specific trainingdata. This makes them suitable for addressing text classification problems fordomains with limited amounts of annotated instances. However, existing researchis limited in scale and lacks understanding of how text generation modelscombined with prompting techniques compare to more established methods for textclassification such as fine-tuning masked language models. In this paper, weaddress this research gap by performing a large-scale evaluation study for 16text classification datasets covering binary, multiclass, and multilabelproblems. In particular, we compare zero- and few-shot approaches of largelanguage models to fine-tuning smaller language models. We also analyse theresults by prompt, classification type, domain, and number of labels. Ingeneral, the results show how fine-tuning smaller and more efficient languagemodels can still outperform few-shot approaches of larger language models,which have room for improvement when it comes to text classification.