Abstract
Adversarial information operations can destabilize societies by underminingfair elections, manipulating public opinions on policies, and promoting scams.Despite their widespread occurrence and potential impacts, our understanding ofinfluence campaigns is limited by manual analysis of messages and subjectiveinterpretation of their observable behavior. In this paper, we explore whetherthese limitations can be mitigated with large language models (LLMs), usingGPT-3.5 as a case-study for coordinated campaign annotation. We first useGPT-3.5 to scrutinize 126 identified information operations spanning over adecade. We utilize a number of metrics to quantify the close (if imperfect)agreement between LLM and ground truth descriptions. We next extractcoordinated campaigns from two large multilingual datasets from X (formerlyTwitter) that respectively discuss the 2022 French election and 2023 BalikaranPhilippine-U.S. military exercise in 2023. For each coordinated campaign, weuse GPT-3.5 to analyze posts related to a specific concern and extract goals,tactics, and narrative frames, both before and after critical events (such asthe date of an election). While the GPT-3.5 sometimes disagrees with subjectiveinterpretation, its ability to summarize and interpret demonstrates LLMs'potential to extract higher-order indicators from text to provide a morecomplete picture of the information campaigns compared to previous methods.