Abstract
The recent success of large language models (LLMs) trained on static,pre-collected, general datasets has sparked numerous research directions andapplications. One such direction addresses the non-trivial challenge ofintegrating pre-trained LLMs into dynamic data distributions, task structures,and user preferences. Pre-trained LLMs, when tailored for specific needs, oftenexperience significant performance degradation in previous knowledge domains --a phenomenon known as "catastrophic forgetting". While extensively studied inthe continual learning (CL) community, it presents new manifestations in therealm of LLMs. In this survey, we provide a comprehensive overview of thecurrent research progress on LLMs within the context of CL. This survey isstructured into four main sections: we first describe an overview ofcontinually learning LLMs, consisting of two directions of continuity: verticalcontinuity (or vertical continual learning), i.e., continual adaptation fromgeneral to specific capabilities, and horizontal continuity (or horizontalcontinual learning), i.e., continual adaptation across time and domains(Section 3). We then summarize three stages of learning LLMs in the context ofmodern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP),and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview ofevaluation protocols for continual learning with LLMs, along with the currentavailable data sources (Section 5). Finally, we discuss intriguing questionspertaining to continual learning for LLMs (Section 6). The full list of papersexamined in this survey is available athttps://github.com/Wang-ML-Lab/llm-continual-learning-survey.