Abstract
The standard way to study Large Language Models (LLMs) with benchmarks orpsychology questionnaires is to provide many different queries from similarminimal contexts (e.g. multiple choice questions). However, due to LLMs' highlycontext-dependent nature, conclusions from such minimal-context evaluations maybe little informative about the model's behavior in deployment (where it willbe exposed to many new contexts). We argue that context-dependence(specifically, value stability) should be studied a specific property of LLMsand used as another dimension of LLM comparison (alongside others such ascognitive abilities, knowledge, or model size). We present a case-study on thestability of value expression over different contexts (simulated conversationson different topics) as measured using a standard psychology questionnaire(PVQ) and on behavioral downstream tasks. Reusing methods from psychology, westudy Rank-order stability on the population (interpersonal) level, andIpsative stability on the individual (intrapersonal) level. We consider twosettings (with and without instructing LLMs to simulate particular personas),two simulated populations, and three downstream tasks. We observe consistenttrends in the stability of models and model families - Mixtral, Mistral,GPT-3.5 and Qwen families are more stable than LLaMa-2 and Phi. The consistencyof these trends implies that some models exhibit higher value-stability thanothers, and that value stability can be estimated with the set of introducedmethodological tools. When instructed to simulate particular personas, LLMsexhibit low Rank-Order stability, which further diminishes with conversationlength. This highlights the need for future research on LLMs that coherentlysimulate different personas. This paper provides a foundational step in thatdirection, and, to our knowledge, it is the first study of value stability inLLMs.