Hard measurement of Soft Skills

Several surveys of U.S. employers identify lack of soft skills as the area where young job-seekers have the largest deficiency .  Work by Heckman and others have provided growing evidence that non-cognitive or soft skills are important for a range of life outcomes.  As a result, a growing number of youth programs have incorporated a soft skills training component – examples include the entra 21 program in 18 Latin American countries, the Jordan NOW program, and the Juventud y Empleo program in the Dominican Republic.

But how do we measure what soft skills youth have? One approach has been to rely on a range of survey questions. Examples from a range of different surveys fielded in the US are given here, and include Likert scales for questions like “I can work with someone who has different opinions than mine” and “I keep going with work even when it takes longer than I thought it would”. The World Bank STEP skills measurement exercise employs such an approach in multiple countries, measuring personality traits, grit, and behavior skills.

However, one might be concerned about the subjectivity of some of these reports, and the extent to which people give socially desirable answers. The question then is whether one can have more objective measures of key soft skills.

Hard measurement
In a short paper I am presenting today at the ABCDE conference, Matthew Groh, Tara Vishwanath and I discuss the use of psychometric testing to reduce labor market asymmetries in Jordan. We worked with Dr Marwan Al-Zoubi, a Jordanian psychology professor to design a number of tests for potential job seekers. One set of these tests were for soft skills measurement.
Soft skills were measured by three interactive exercises:
 

1. Group Exercise:  The group discussion was centered on the design of an amusement park. Five to eight participants were tasked to redesign a failing amusement park in Jordan. Each participant was given a pre-defined role in marketing, HR, finance, customer service, and management with specific responsibilities and often conflicting goals. By creating an opportunity for job candidates to interact in a group in a structured manner, evaluators could evaluate how job candidates work in groups. Two trained evaluators evaluate each candidate on 10 soft skill categories during the group discussion, tabulated below:

2. Role-playing exercise: a one on one exercise with the evaluator and the job candidate that is intended to test the candidate under pressure. The job candidate plays the role of a customer service associate and the evaluator plays the role of an angry customer who just bought a computer that broke down the past night. The job candidate has to calm down the customer and come up with a solution within the framework of the companies’ rules. This component is assessed by one evaluator on soft skill components 1 through 5 in the table above.

3.  Skills-based interview: where the evaluator asks questions to elicit examples of leadership, teamwork, and overcoming obstacles. This last component was assessed by one evaluator on the soft skill components 6 through 10 in the table above.The group exercise lasted 40 minutes, and the individual exercises lasted 5 to 10 minutes each. We then average the scores in each of the 10 soft skill categories and form a principal component of the average scores in the 10 soft skill categories to create a soft skills index.

We did this for 1567 recent graduates of universities and community colleges in Amman, Jordan. 58 percent of the sample were female, average age was 23, and the most common majors were accounting and business, engineering, and computing and IT.

Does this predict employment?
A follow-up survey that took place on average 10 months after the soft skill assessment suggests that this exercise is capturing something that matters to employers.
 

• For females, employment was 41.7% for those in the bottom quartile of soft skills, versus 55.3% for those in the top quartile.
• For males, employment was 54.5% for those in the bottom quartile, compared to 64.7% for those in the top quartile.
• Conditional on being employed, both males and females earn approximately 30 percent more per month if their soft skills are in the top quartile vs the bottom quartile.

Of course we might be concerned that soft skills are correlated with a whole lot of other characteristics that also matter for employment. In the paper we show that they continue to matter for predicting earnings even after conditioning on mental ability, excel ability, English test score, Big-5 personality scores, the student’s score in the national end of high school exams, the university they went to, their major, their previous work history, and their marital status.
So it seems that this measure is picking up something that both seems reasonable in spirit, and seems to have useful predictive power for employment in practice.

When will this be most useful?
Clearly this is a non-trivial measurement exercise, taking a full hour per person, which is as long or longer than many surveys. It requires trained staff to do the assessments, and requires bringing several people together in a group for the group assessment. So it is not an easy substitute for a set of self-reported questions.
It therefore seems most useful for the following contexts:
 

• Interventions that teach soft skills training – ideally one would like to measure whether these programs actually improve soft skills, before then tracing their impacts on ultimate outcomes like employment.
• As a way of measuring soft skills to help in matching workers to jobs – our ongoing work in Jordan looks at an intervention which does this.
• In contexts like some youth programs where you have people coming into workshops or several day sessions anyway, in which the logistics may be easier.

Anyone else hit upon good ways to measure soft skills objectively? This is an area where the existing literature is rather slight, so there are some interesting openings for new research.