The hunt for the elusive Big Five

While there is a large body of evidence on the importance of cognitive ability for predicting social and economic successes, personality traits such as Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism—also known as the Big Five—are often emphasized to be equally important for many socioeconomic outcomes. Ample empirical evidence from the US and other high-income countries shows that the Big Five correlate with earnings, employment and other labor market outcomes. Recent reviews conclude that Conscientiousness and Emotional Stability in particular, are strong predictors of job performance and wages. Furthermore, personality traits are thought to be particularly important for people with lower levels of education or job complexity, whereas cognitive ability is more important at higher levels of job complexity. One may hence believe that personality traits could be even more important in low and middle-income countries, where large shares of the population participate in lower-complexity jobs.

Unfortunately, there are several limitations for measuring personality traits, especially in developing contexts. In a recent study on skills and agricultural productivity which uses methodological research conducted in partnership with the World Bank’s Living Standards Measurement Study (LSMS) with funding from the Department for International Development (DFID), Laajaj and Macours found a significant amount of non-classical measurement errors on a wide range of non-cognitive skills measures in Kenya and Colombia. This raises questions about the validity and reliability of related personality trait measures in developing country contexts.

In their new paper “Challenges to Capture The Big Five Personality Traits in Non-WEIRD Populations”, Laajaj, Macours, and coauthors look at the measurement, validity, and reliability of personality traits outside Western Educated Industrialized Rich and Democratic (WEIRD) populations. For that purpose, they analyzed 29 face-to-face surveys from 94,715 interviewees in 23 non-WEIRD populations and 198,365 self-selected respondents of internet surveys from the same countries. Evidence from this study shows that in most databases, commonly used personality questions generally do not accurately measure the intended personality traits and show low validity. Here are some key findings:

• The common wisdom that Big Five questions accurately capture Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism does not hold for most survey data in low and middle-income countries. Even though the Big Five factor model may well be universal, it appears hard to uncover such factor structure with survey data in contexts other than those for which it was developed. The lack of support for the Big Five factor model across a large set of face-to-face surveys in diverse contexts indicates that the issues identified are not unique to a specific data collection exercise but point to a general problem in the measure of personality traits through survey data in developing countries.
   
• Many questions correlate more with items intended to measure a different personality trait than with items intended to measure the same personality trait. This makes it arguably hard to interpret items as capturing the intended personality trait. A set of items meant as proxy for a specific personality can capture other factors and lead to incorrect inferences about the relationship between specific personality traits and income (or other outcomes of interest).
   
• Internal consistency for all survey data is substantially lower than for the internet data and the US data. The differences found between the internet data and the survey data suggest that cultural differences are unlikely to be the main driver of low validity.
   
• The World Bank's Skills towards Employment and Productivity (STEP) data shows that there is a positive correlation between measures of cognitive ability and measurement error. Moreover, respondents with lower cognitive skills are more likely to agree to contradictory statements, referred to as acquiescence bias (or yea-saying).
   
• Enumerators explain up to 25% of the variation in some regions, indicating that interviewers influence responses in very substantial ways. Response bias can be reduced by i) balancing reversed and non-reversed items and using acquiescence bias correction, ii) using self-administrated surveys, iii) improving the quality of translations (especially for respondents with lower cognitive ability) and iv) randomizing the assignment of enumerators.

This work hence highlights the risk of misinterpreting Big Five survey data and provides a warning against naïve interpretations of personality traits without evidence of its validity. Before using questions aimed at measuring personality traits, data validation in each context is a must, especially for data from low and middle-income countries.  Inferences about a specific personality trait are only credible if the data has clear structure and good reliability and validity. Overall, the findings from this study call for the development of innovative methods that more accurately capture specific personality traits in developing country surveys, which would need to focus on reducing the impact of response patterns and enumerator interactions and be more adapted to the context and mode of administration.