Are Non-Cognitive Gains in Education More Important than Test-Scores?

|

This page in:

Most educational interventions are widely considered successful if they increase test-scores -- which indicate cognitive ability. Presumably, this is because higher test-scores in school imply gains such as higher wages later on. 

However, non-cognitive outcomes also matter---a lot.

In the graph above, from James Heckman's 2007 presentation, "Investing in Disadvantaged Young Children Is Good Economics and Good Public Policy," are IQ scores measured on the Stanford-Binet Intelligence scale -- for the treatment group and the control group for the children that took part in the Perry Pre-School project. [See above, from Figure 8a in Heckman's presentation by Terman & Merill, 1960]. This study examines the lives of 123 African-Americans born in poverty and at high risk of failing in school. From 1962–1967, at ages 3 and 4, the subjects were randomly divided into a treatment group that received a high-quality preschool program and a comparison group that received no preschool program. As the graph above clearly shows, by age 10, IQ scores in both treatment and control groups looked the same.

Here are the economic effects at age 27 from the Perry Pre-School intervention below:

Source: from Figure 8c in Heckman's presentation by Barnett, 2004.

In the study's most recent phase, 97% of the participants still living were interviewed at age 40. Additional data were gathered  from the subjects' school, social services, and arrest records.

The study found that adults at age 40 who participated in the Perry Pre-School program had higher earnings, were more likely to hold a job, had committed fewer crimes, and were more likely to have graduated from high school than adults in the control group who did not participate.

While measuring and estimating the impact of educational interventions on non-cognitive and behavioral outcomes is now mainstream in the US
(see: Cullen, Jacob and Levitt in Econometrica), it has received less attention in low-income countries.

Possibilities for why Non-Cognitive Outcomes have been Measured Less in Developing Countries:

One argument are that test-scores are very weak in many low-income countries and at those very low levels of learning, improving test-scores should remain the #1 aim of the schooling system - and non-cognitive outcomes can wait until students can read and write. This argument doesn't convince me.

A new paper by Pramila Krishnan and Sofya Krutikova, entitled, "Skill Formation in Bombay’s Slums," measures psychosocial skills (self-esteem and efficacy) among poor households in Mumbai found the following:

  1. Psychosocial skills directly correlated with wages and additional controls for cognitive outcomes have very little impact on the wage psychosocial skills association (see the graph below).
  2. These skills can be improved through appropriate interventions---the authors' evaluation of such a program in Mumbai suggests that the “effect of the intervention on these (psychosocial) measures is between 0.75 and 0.90 of a standard deviation.” That’s an enormous increase in wages (between 10 and 20 percent) according to graph below.

 

Further, while cognitive outcomes tend to fade-out very quickly (see a summary of fade-out in test-scores in Do Value-Added Estimates Add Value? Accounting for Learning Dynamics), non-cognitive improvements persist and indeed may become more salient over time.


What is the Takeaway?

It's hard to say what countries should focus on. But clearly, we should be thinking a lot more about a full package of cognitive and non-cognitive outcomes rather
than test-scores alone. Let me know what you think........

Authors

Jishnu Das

Professor, Georgetown University

Paul B
November 18, 2010

And yet, of course, as we encourage performance based budgeting everywhere, we are constantly looking for things that can be measured in school - & the non-cognitive benefits are extraordinarily difficult to measure. As is genuine learning as compared to short-term memory fact cramming.

It would be helpful if the World Bank or someone could develop, and give a lot of publicity to, guidance on how to set useful targets in primary school for use with performance budgeting.

Lin
November 05, 2010

Couldn't agree more. The whole obsession with test scores frustrates me so much. I was a pretty smart kid and I was so bored and not challenged in my "top in the nation" high school, so I kept dropping out. Once, i came back and had to take classes with the grade below me, which was also the first year of a new standardized test, the OGT. I had taken the classes before, and the difference in the same classes, with the same teachers was amazing, in the worst way possible. I didn't see (or experience) any learning, only cramming useless facts into our short term memories. My US history teacher had been my favorite, we'd act out immigration and all sorts of things, really giving us a feel for the human face of history and why things happened as they did. He had almost no activities like that once the school started pushing OGT studying. Needless to say, I quit again.

I received a full scholarship to college, so I'm no worse for the wear, but I feel sick whenever I hear people pushing test scores. To me, high test scores mean almost nothing. What good are dates and names that they won't remember 2 weeks after the test? I'd much rather see kids learning concepts and critical thinking, learning to appreciate and enjoy science and learning, and to learn skills that will be of use to them in their lives and career paths, etc. Stuff that will stick. Tests do not!

Jishnu Das
November 19, 2010

It may be useful to separate out the descriptive facts about non-cognitive aspects of education (they matter and they are malleable, even in very low-income settings) from the policy implications of these facts. One problem is that the results in this post are based on small scale studies, with dedicated and motivated teams. The results of an evaluation of the large early childhood scheme in the U.S., Head Start, is a mixed bag (http://www.acf.hhs.gov/programs/opre/hs/impact_study/reports/impact_stu…). So, I am not sure whether we have a proven to-scale program that improves non-cognitive outcomes among kids (I am not an expert, so would love to hear back on this).

On what we should measure, I have been deeply influenced by Ed Lazear's analysis of the problem of high stakes testing (http://www.mitpressjournals.org/doi/abs/10.1162/qjec.121.3.1029?journal…). Lazear's point is that when the cost of learning is high and the cost of monitoring is also high, its best to test with high stakes--announcing the test concepts beforehand. The intuition is simple and brilliant--if the cost of learning is super high, then the incentives from low-stakes testing are too low to result in any improvements. With high-stakes testing, the kids will focus on the concepts that are to be tested, but this may increase learning, because at least those concepts will be learnt.

To put it another way, suppose I want to test tomorrow. I can say "I am going to test you with 8 questions, 2 each on addition, subtraction, division and multiplication" and you will get a dollar for every question you do right". Or, I can say "I am going to test you on 8 single-digit addition questions" and give you a dollar for every question you get right. Given that as a student, I have one night to study, in the first scenario, I may well decide not to study at all if I can learn only one concept, since the maximum I can get is $2 and the cost of studying is (say) $4. But in the second (high-stakes) test, I will study single-digit addition, because I stand to gain up to $8. So at least I will end up learning something in the second case. A contrived example---the paper is much nicer.

From the comments, what I hear is that this is going to be costly to measure and potentially costly to change. Which is disappointing, since if you believe Lazear's line of reasoning, it suggests that this is something that school systems should not invest in currently. Not because non-cognitive aspects of education are unimportant, but because it will be very hard to provide appropriate incentives to improve them in a scaleable manner.

Having said that, if you have 12 minutes, see this video, which argues that a change in the focus of schooling is critical:

http://www.youtube.com/watch?v=mCbdS4hSa0s

& thanks for the many enlightening comments.

Alexandria Valerio
November 08, 2010

Indeed, very interesting research findings. The HD Anchor is currently in the process of launching a multi-country study to measure the cognitive, non-cognitive and technical skills in the working age population, as well as surveying employers to determine the extent to which they desire and value different type of skill sets when making hiring decisions. We have many challenges, ranging from ensuring skill measurements are international comparable, making decisions on whether to focus only on non-cognitive skills that are malleable, and, of course, disentangling issues of causality. We hope our research findings will shed light on skill mismatches and on the relative importance of different type of skills sets in the labor market.

Helen Abadzi
November 07, 2010

Self-efficacy and self-esteem are merelly estimates of earlier success (i.e. bayesian statistics). Success derives from knowledge and skills, which people must use effortlessly and appropriately. Tests are used to assess these. And test performance attests to their fluency.

Certainly events may depress people's perceived ability of whether they can accomplish something, but inflating their self-esteem or self-efficacy artificially may not make them perform better in the labor market. US schools have done this with poor results. (Jishnu's team should find the studies.) Students may start feeling entitled, but without the skills to reap the rewards they expect.

All of us have seen charming and self-confident people who promise a lot more than they deliver. Is this the type of employee that should be promoted in the labor markets of the world? It would be useful to learn more from the volumes of research on the psychological concepts used in this blog before settling on solutions.

James Gresham
November 08, 2010

These findings are very interesting. We know that an individual's success later in life is affected by a range of factors, including but not limited to cognitive ability. For this reason, I'm also surprised by the high level of attention paid to test scores, which may or may not accurately reflect cognitive ability anyway. The findings here suggest several things to me. First, it seems clear that we need to expand our definition of "success" with regards to educational interventions to include non-cognitive as well as cognitive outcomes, especially in developing countries. Perhaps this could involve some link between the two types of outcomes. For example, if we could identify which types of cognitive skills have the greatest positive impact on non-cognitive outcomes, and we could design tests to measure these skills, then test scores may be more accurate indicators of “success” later in life.

Another takeaway from these findings is the powerful effects of intervening early in a child's life. The Perry Pre-school program is one of the most studied ECD programs in the world. Evidence on this program, and other programs around the world, identifies a clear positive relationship between early childhood interventions and various educational, social, and economic outcomes throughout life. Many of these outcomes persist over time, so in my perspective, there is a clear rationale for continuing and expanding investments in this area.

Mohamad Sbeiti
December 14, 2010

I have been working for two years on an education development program for a private school in lebanon. the school is located in the poor suburbs of lebanon. the project aimed to enhance the quality of education, among other non educational goals such as improving the processes and procedures of education supervision, restructuring the school organization...etc

During the course of my work, I came upon a case where one parent complained about a reading and comprehension exam. she said that this exam is above the level of her kids class. We investigated her claim and found that the teacher used to rehearse her students on the answers before the exam date through hinting to them the answers. The students would then memorize the answers and reiterate them during the exam. We found that all students were not actually reading, they were merely trying the figure out during the exam where to place their pieces of memory on the answer sheet. If the teacher changes the sequence of questions or change some key words, the kids will be totally lost and score less. we also found that high perfroming kids are masters at this game. they, with the help of their parents and under their instructions, rehearse better at home.

The biggest challenge facing the school is how to minimize the drop out rate of students. Students were dropping out for many reasons such as, dysfunctional families, low income and poor academic performance. As the school has no influence over the external environment, the school decided to focus on motivating poor performing students.

the reading and comprehension exam episode openned the eyes to the fact that it is impossible to test kids in lower elementary for the acquired skills. skills are slow to develop and they might not appear until later stages on ones life such as the case of the Heckman's study.

As a result, we changed the evaluation system for the elementary stage. we made it more flexible and more subjective to the teachers personal judgement, since the teacher is in direct contact with the students and more knowledgable about their cognitive and non-cognitive performance. regular testing methods are not very reliable; that teacher tried to outsmart the testing limitation to improve her results since the school administration and most parents care about grades. Parents use the grades to measure the return on their investment in their kids education, even in elementary school due to their limited access to money. We stressed less on exams and more on teaching in class. Also, we changed the report card to become more readable by kids themselves. the main goal of the new evaluation system is to reward effort and motivate the students to study rather than demotivate them with low grades on skills they didnt acquire yet.

BEN
February 15, 2012

In all honesty I truly catch your rift, to me testing has its advantages but it also has its disadvantages, it hinders creativity, which is the thing that mostly needed to in the "practical" world, not a world where your boss asks you to fill out an OMR sheet to solve a problem in the company. This is why so many youths who are so intelligent think that they are not. Thanks again for the post!
Cheers,
Ben A .