Abstract
Transformer language models (LMs) are fundamental to NLP researchmethodologies and applications in various languages. However, developing suchmodels specifically for the Russian language has received little attention.This paper introduces a collection of 13 Russian Transformer LMs, which spansencoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder(ruT5, FRED-T5) architectures. We provide a report on the model architecturedesign and pretraining, and the results of evaluating their generalizationabilities on Russian language understanding and generation datasets andbenchmarks. By pretraining and releasing these specialized Transformer LMs, weaim to broaden the scope of the NLP research directions and enable thedevelopment of industrial solutions for the Russian language.