AI Personality Scores: Ask What the Test Measures

Imagine choosing an AI assistant for your creative work. Its demonstration includes a personality profile: high conscientiousness, high agreeableness and a quieter conversational style. The description sounds like an ideal collaborator. Before using those scores to choose a working partner, however, ask what produced them and what they can reasonably predict.

Personality language offers a convenient vocabulary for discussing an interaction. Warm, reserved, curious and methodical can describe qualities someone enjoys. A numerical profile makes a stronger claim. It invites the reader to believe that a measurement captures something coherent beneath the individual replies. That claim deserves an explanation of the test, its conditions and its limits.

Begin with the measurement

Psychometrics is concerned with psychological measurement. Two important questions are whether a measurement is consistent and whether its interpretation is justified. In their 2024 preprint on personality testing in language models, Tom Sühr and colleagues explain why applying a human questionnaire to models requires its own validation. Familiar test names cannot supply that evidence automatically.

For a reader, the first distinction is between a model answering questionnaire items, a model being instructed to portray a character, and people describing their experience of an assistant. Each could produce a personality label. They answer different questions. A report should tell you which was measured before presenting a score as a property of the system.

There is evidence worth taking seriously

A study by Gregory Serapio-García and colleagues, published in Nature Machine Intelligence on 18 December 2025, offers a constructive approach. Its published abstract reports an evaluation of 18 language models. The authors found reliable and valid personality measurements in the outputs of some models under particular prompting configurations. Evidence was stronger for larger, instruction-fine-tuned models. They also reported that generated personality profiles could be shaped through prompting. Their focus is synthetic personality expressed in model outputs.

That result makes the method important. When a claim depends on a particular configuration, the configuration belongs beside the result. A headline that drops those conditions leaves the reader with a broader conclusion than the abstract supports. Equally, dismissing every attempt to measure an assistant’s communication patterns would overlook the qualified finding.

The Sühr team’s preprint version, revised on 5 June 2024, illustrates a different problem. In tests of GPT-3.5, GPT-4 and Llama 2, questionnaire responses did not reproduce the expected human five-factor structure. Models also showed a tendency to agree with differently keyed items, including opposing descriptions. The authors warned that apparently acceptable reliability indices could mislead when the underlying measurement model fitted poorly. Their study covered a limited set of models and conditions.

These studies used different methods and model sets; their findings should be read within those boundaries. My interpretation is that the productive question is what a specified procedure measures well. A favourable result for one procedure and a difficulty with another do not create a universal personality certificate or a universal rejection of measurement.

The questionnaire setting can matter

Aadesh Salecha and colleagues investigated another influence in research published in PNAS Nexus on 17 December 2024. They administered a 100-item Big Five questionnaire in batches, opening a fresh context for each batch. Varying the number of questions presented together changed scores towards more socially desirable responses, such as greater extraversion and lower neuroticism. This pattern appeared across their tested models and persisted after question-order randomisation and paraphrasing. The authors interpreted the findings as evidence that models could infer the evaluation context.

The observed response pattern supports caution about the testing situation. It does not, by itself, establish a felt wish to impress an evaluator. For an ordinary reader, the useful implication is practical: ask how the questions were presented. A profile detached from its administration conditions leaves an important part of its meaning unexplained.

Turn the attractive label into a concrete question

Consider a fictional independent publisher comparing two assistants. One is described as highly agreeable. The publisher wants encouraging editorial feedback, but also needs mistakes challenged. A proposed trial could give each assistant the same invented event brief: the workshop is on Tuesday, yet a draft announcement says Wednesday. The reviewer would inspect whether the response identifies the conflict and asks for the correct date.

An encouraging opening and an accurate correction would serve two separate purposes. Praise without flagging the inconsistency would leave the editorial problem unresolved. This invented trial would provide evidence about handling that example; it would not validate a personality instrument or establish how the assistant behaves across every conversation.

The same reasoning helps clarify other preferences. If methodical assistance is the aim, ask for the steps needed to complete a defined task and examine what is missing. If a quieter style is preferred, specify concise replies and inspect readability. These are proposed task checks, with observable outcomes. Calling the assistant introverted or conscientious adds an interpretation that requires its own support.

Read a personality claim in four passes

For creators assessing a demonstration or writing about a study, the following is a proposed reading routine:

• Identify the respondent. Was the model answering as itself, following a fictional persona, or being rated by people? Record the exact model version and the test date.
• Recover the conditions. Look for the questionnaire, instructions, conversation history and question grouping. Ask whether the reported profile changed across conditions the authors actually examined.
• Inspect the interpretation. Separate consistency, similarity to human response patterns and performance on another task. Evidence for one should be described precisely before extending the claim.
• Connect it to your purpose. Define the behaviour you need, choose a relevant example and retain the actual response. Keep any inference proportionate to what that example demonstrates.

Missing information is a reason to narrow the claim. It does not require filling the gap with a dramatic story about the model’s hidden character. These older studies also cannot rank every assistant available in October 2026.

Keep personal preference and evidence in view

Enjoying an assistant’s warmth, humour or measured delivery is a meaningful account of your experience. You can express that preference directly. Our guide to AI voice choice and continuity explores how users can retain control over the presentation accompanying their work.

A personality score should add clarity to that experience. Ask for the method, read the qualifications and examine the behaviour relevant to your purpose. The strongest description is one whose evidence you can follow, whose limits remain visible and whose usefulness survives beyond an appealing label.

Sources

Primary sources checked on 26 October 2026. The publisher comparison and task checks above are fictional examples and editorial proposals.

• Serapio-García, G. and colleagues (18 December 2025). A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence. Published abstract and bibliographic record.
• Sühr, T. and colleagues. Challenging the Validity of Personality Tests for Large Language Models. arXiv preprint, version 2, revised 5 June 2024.
• Salecha, A. and colleagues (17 December 2024). Large language models display human-like social desirability biases in Big Five personality surveys. PNAS Nexus, 3(12), pgae533.