Building Grit in the Classroom and Measuring Changes in it

About a year ago I reviewed Angela Duckworth’s book on grit. At the time I noted that there were compelling ideas, but that two big issues were that her self-assessed 10-item Grit scale could be very gameable, and that there was really limited rigorous evidence as to whether efforts to improve grit have lasting impacts.

A cool new paper by Sule Alan, Teodora Boneva, and Seda Ertac makes excellent progress on both fronts. They conduct a large-scale experiment in Turkey with almost 3000 fourth-graders (8-10 year olds) in over 100 classrooms in 52 schools (randomization was at the school level, with 23 schools assigned to treatment).

The intervention
Teachers in the treatment schools received training on how to adopt the idea of grit as part of their overall teaching philosophy, including a shift to praising effort and attitude towards effort rather than just outcomes, and provided with material including animated videos, mini case studies, and classroom activities. Typically teachers received a minimum of 10 sessions of training, lasting 12 weeks. The material emphasized: i) the plasticity of the human brain rather than the notion that ability is fixed and innate; ii) the role of effort in enhancing skills and achieving goals; iii) the importance of interpreting failures in a constructive way to learn from; and iv) the importance of goal setting. Students are encouraged to set goals, exert effort to work towards these goals, and to avoid interpreting immediate failures as a lack of innate ability or intelligence – leading to the first part of the paper’s title “Ever fail, try again, succeed better”.

Measuring Grit
A key innovation in the paper is then to develop and use an incentivized real effort task to measure grit, rather than just relying on student self-reports. The students were given a grid of numbers, and had 1.5 minutes to find three pairs of numbers that each summed to 100. They played the game in five rounds, and before each round had to choose between an easy task (left picture below) and hard task (right picture). After each round is completed the experimenters check their sheets and circle either “succeeded” or “failed”, before the choice for the next round is made. The easy task yielded 1 prize in the case of success, and 0 for failure, and almost everyone was able to do it. The hard task yielded 4 prizes in the case of success, and 0 for failure, with only about one-quarter of students getting this right.

For the first round, fifty percent of the students are randomly assigned to the difficult task, while fifty percent get to choose. Then for all subsequent rounds individuals get to choose which game to play round by round. They find treatment has no impact on performance in the first difficult game, which is nice, because they can then look at those who failed, and see who choose to try again.

Then a final nice addition is to tell the students that they will come back in 1 week to play the game again, and that they have to choose now which game they will play in a 1 week, but that they will have access to an exercise book to practice the difficult questions in between if they want. As in the first round, 50 percent are randomly assigned to the hard game at follow-up, while 50 percent get their choice.

Results
Impacts on Grit

• In the first round, more than 70 percent of students in the control group choose the difficult game, but this drops off through the rounds, until only 27-41% attempt it in the fifth round. Treated kids are less likely to give up on a hard task – they are 9-13 percentage points more likely to choose the difficult game in all 5 rounds.
• Of those randomly assigned to the difficult game in the first round, treated students are 14-18 percentage points more likely to persevere with the difficult task after failure.
• Treated students are 14-19 percentage points more likely to plan to take the difficult game in one week, and when the difficult task is randomly imposed on them, are 9-11 percentage points more likely to succeed one week later.

Impacts on test scores
The authors also administer a standardized math test and Turkish test. Treatment increases performance in the math test by 0.28 standard deviations, and on the Turkish test by 0.13 standard deviations. This test is done 4 months after the intervention was implemented.

Can you exhibit too much grit?
This was another point that came up in the discussion around the book Grit when it came out. In this context, the question is whether the students are profit-maximizing to persevere with a difficult task that has lower odds of success, or rational to give up? The authors find a positive and significant effect on the total payout students get, and then tries to predict their individual profit-maximizing decisions given baseline characteristics and finds treated students are more likely to choose the profit-maximizing choice in later rounds.

Caveats/Critique
I very much like the measurement task here and the evidence that grit can both be taught and make a difference on outcomes we care about like test scores. Some things to consider:

• The outcomes are very short-term, so there is a question as to whether this yielded a lasting impact on changes in mindset for the students. This is with 4^th graders, so there needs to be a long persistence for it to affect final education outcomes and eventually labor earnings. Sule tells me that they are currently revising the paper, and have now tested one of the two samples they use 2.5 years later, and the impact on math scores persists (0.18 s.d), but the effect on the verbal score seems to have faded.
• The paper does not contain any information on costs, but does note that it is a low-cost intervention.
• The intervention is randomized at the school level, and split into two samples which are separated for the analysis. One sample has 16 schools (8 T and 8 C). Standard errors are clustered at the school-level, so you might worry about clustered errors with small samples. Sule tells me the revision shows the results survive using permutation tests.
• The exercise and analysis are only for students who were present in class on the days of the exercises. There is no discussion of what absence rates are like or whether we need to worry that the intervention had any impact on this. Sule tells me non-attendance rates are 20% on any given day, but is balanced across treatment status.

I definitely look forward to seeing the revised version of the paper, and this intervention seems worth replicating and extending in other contexts. It also seems interesting to think about how you would modify this incentivized grit measure task to use with adult populations.