- GPT-4 predicted personality scores months before any human ever answered a question
- A Bosch oven manual was enough for GPT-4 to invent personality traits
- DSM-5 predictions hit 0.71 accuracy; astrology predictions somehow hit 0.85 accuracy
Human personality has traditionally been measured by asking people questions and comparing their answers across established psychological frameworks.
A new study published on iScience suggests GPT-4 can go further by estimating how people collectively respond to personality questions before answering begins.
Researchers Rotem Monsa, Aviv Zohar, and Shahar Arzy of the Hebrew University and Hadassah Medical School tested whether the model could infer patterns in human responses from language alone, rather than merely generating plausible questionnaire wording.
Latest Videos FromTechRadar
GPT-4 predicted responses before humans answered
The team used GPT-4 to create personality questionnaires from two very different textual sources.
One questionnaire drew on the DSM-5, the standard clinical manual used worldwide by psychiatrists to diagnose personality disorders with precision.
The other version pulled statements from an astrology textbook that assigns particular personality traits to each of the twelve zodiac signs.
“We wanted to choose texts that describe human personality in very rich detail but also sit on opposite ends of a spectrum in terms of scientific grounding,” said Rotem Monsa, lead author of the study.
“The DSM-5 was refined through decades of clinical research and is the standard diagnostic manual in clinical psychiatry, known worldwide.”
Both questionnaires were then given to 600 participants alongside the Big Five Inventory, the most validated personality measurement tool available.
Before collecting those responses, GPT-4 estimated average scores for individual questions using a five-point Likert response scale.
This gave the researchers a way to compare predictions against subsequently collected evidence.
The model’s estimates correlated strongly with actual participant averages, reaching 0.71 for DSM-derived questions and 0.85 for astrology-derived questions.
This decent internal consistency for DSM-5-based questionnaires suggests that traits that typically appear together in real life also correlated in responses.
The astrology questionnaire, by comparison, showed a noticeably weaker internal consistency across most of the personality traits it measured overall.
GPT-4 also estimated relationships between individual questions, producing correlations of 0.74 and 0.69 for DSM and astrology questionnaires, respectively.
The researchers therefore argue that GPT-4 appears capable of capturing population-level response tendencies embedded within language and questionnaire content.
However, they specifically caution that predicting average responses does not establish that the resulting questionnaire accurately measures individual personality differences.
Testing the limits with an oven manual
The researchers also tested whether GPT-4 would generate personality-style questions when given material containing no explicit personality information.
They supplied the model with a Bosch oven manual and a literary description of a landscape from The Lord of the Rings.
GPT-4 still produced questionnaire items resembling personality assessments, even though neither source was designed around psychological traits.
That result suggests the model has learned strong associations between language patterns, personality descriptions, and familiar questionnaire structures.
Yet the researchers found that unrelated source material could produce items that were semantically weak, narrow, or poorly connected.
This distinction matters because a convincing question does not necessarily measure the psychological characteristic that researchers intended to examine.
The study also found important differences between the questionnaires when researchers examined their underlying statistical structures.
The Big Five questionnaire most closely matched its expected organization, while the DSM and astrology questionnaires showed weaker confirmatory factor-model fits.
Astrology-derived items also failed to preserve their intended elemental structure, instead forming broader patterns resembling established personality dimensions.
The results nonetheless showed comparable predictive performance across many life outcomes, although astrology-based measures performed worse for mental-health outcomes.
For now, the evidence supports a narrower claim: GPT-4 can anticipate aggregate human response patterns from textual information with measurable accuracy.
“Given that personality traits are reflected in language, LLMs may have learned the structure of human personality as a natural byproduct of their training,” Monsa added.
“So, while they were not taught specifically psychology or personality theories, these are already embedded in the language that LLMs learn from.”
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.
https://cdn.mos.cms.futurecdn.net/xs9FSbfR2xN6s5PKQanXm3-1920-80.png
Source link
