Abstract
While supervised fine-tuning (SFT) has been a straightforward approach fortailoring the output of foundation large language model (LLM) to specificpreferences, concerns have been raised about the depth of this alignment, withsome critiques suggesting it is merely "superficial". We critically examinethis hypothesis within the scope of cross-lingual generation tasks, proposingthat the effectiveness of SFT may be constrained by its reliance on priortokens to guide cross-lingual generation. Based on this crucial insight, and inresponse to the challenges posed by the costly and limited availability ofnon-English data for SFT, we introduce a novel training-free alignment methodnamed PreTTY, which employs minimal task-related prior tokens to bridge thefoundation LLM and the SFT LLM, achieving comparable performance withouttraining. Experiments on machine translation and part-of-speech tagging acrosseight languages demonstrate the efficacy of PreTTY in cross-lingual settings.Remarkably, by initiating the decoding process with only one or two priortokens, foundation LLMs can achieve performance comparable to their SFTcounterparts. This method presents a cost-effective alternative to SFT andadvances the democratization of multilingual LLMs.