Abstract
Modular deep learning is the state-of-the-art solution for lifting the curseof multilinguality, preventing the impact of negative interference and enablingcross-lingual performance in Multilingual Pre-trained Language Models. However,a trade-off of this approach is the reduction in positive transfer learningfrom closely related languages. In response, we introduce a novel method calledlanguage arithmetic, which enables training-free post-processing to addressthis limitation. Inspired by the task arithmetic framework, we apply learningvia addition to the language adapters, transitioning the framework from amulti-task to a multilingual setup. The effectiveness of the proposed solutionis demonstrated on three downstream tasks in a MAD-X-based set of cross-lingualschemes, acting as a post-processing procedure. Language arithmeticconsistently improves the baselines with significant gains in the mostchallenging cases of zero-shot and low-resource applications. Our code andmodels are available at https://github.com/mklimasz/language-arithmetic .