Abstract
Pre-trained vision-language (V-L) models such as CLIP have shown excellentperformance in many downstream cross-modal tasks. However, most of them areonly applicable to the English context. Subsequent research has focused on thisproblem and proposed improved models, such as CN-CLIP and AltCLIP, tofacilitate their applicability to Chinese and even other languages.Nevertheless, these models suffer from high latency and a large memoryfootprint in inference, which limits their further deployment onresource-constrained edge devices. In this work, we propose a conceptuallysimple yet effective multilingual CLIP Compression framework and train alightweight multilingual vision-language model, called DC-CLIP, for bothChinese and English context. In this framework, we collect high-quality Chineseand English text-image pairs and design two training stages, includingmultilingual vision-language feature distillation and alignment. During thefirst stage, lightweight image/text student models are designed to learn robustvisual/multilingual textual feature representation ability from correspondingteacher models, respectively. Subsequently, the multilingual vision-languagealignment stage enables effective alignment of visual and multilingual textualfeatures to further improve the model's multilingual performance. Comprehensiveexperiments in zero-shot image classification, conducted based on the ELEVATERbenchmark, showcase that DC-CLIP achieves superior performance in the Englishcontext and competitive performance in the Chinese context, even with lesstraining data, when compared to existing models of similar parameter magnitude.The evaluation demonstrates the effectiveness of our designed trainingmechanism.