Teacher Turnover and Student Performance

Selective teacher retention policies are both complex and controversial wherever they’re implemented. In Washington, DC, where I live and work, things are no different: pursuit of such policies arguably led to the early departures of a mayor and the chancellor of the public school system (DCPS). A January 2016 paper by Adnot, Dee, Katz, and Wyckoff evaluates the effect of teacher turnover under on student achievement under IMPACT, DCPS’ performance assessment and incentive system for its teachers – introduced in the 2009-10 school year (NBER WP version, gated, can be found here.

Given the importance of (and the large variance in) teaching quality, policies can employ one or both of two strategies: improve the quality of existing teachers or change the composition of teachers. Obviously, tackling one can have effects on the other. For example, tackling the composition can have an effect on the existing staff to invest in becoming more effective. Alternatively, teacher-training programs can change the composition of turnover and have an effect on teaching quality through both channels.

IMPACT assigns scores to each teacher based on her/his performance at the end of the year, which is a weighted average of five components that are consistent with emerging best-practice principles (e.g. clearly described standards, multiple teacher observations by various observers, and student performance). The grades that are assigned to the teachers can range from ineffective to highly effective, with consequences that can range from dismissal to very large bonuses and salary increases. One could imagine that, in certain labor markets, it might be difficult to replace dismissed teachers with as effective or better ones. If the system has a lot of noise, it could lead to uncertainty and demoralization, leading to the departure of the more effective teachers that it aims to retain. Higher turnover, even if successful in replacing the departed, can itself cause a decrease in student performance through disruption of remaining teachers’ schedules. On the other hand, one can imagine that low-performing teachers can be replaced with better ones while the already high-performing teachers are retained successfully.

Since the reform is system wide, there is no chance for an RCT for the authors, who employ a panel data difference-in-difference (DD) analysis to the data. The lack of reliable teacher data linked with students prior to IMPACT prevents them from extending the DD analysis to “before baseline” and the approach has to make several assumptions regarding sorting of students, of teachers, and unobserved heterogeneity. The authors discuss these shortcomings and how they check the robustness of their findings for these potential confounders. So, what do they find?

• DCPS has a high turnover rate, at around 18% per year, higher than reported for 16 other urban school districts in a recent study. The rate is lower among high-performing teachers (13%), and even lower if those teachers are working in low-poverty schools. In contrast, the annual attrition rate among low-performing teachers is 46%.
• Teacher composition changes for the better: While high-performing exits are hard to replace with teachers with the same IMPACT scores, low-performing exits are replaced with teachers with much higher IMPACT scores that the average IMPACT score among entering teachers is higher than those exiting (see figure below).

• Student outcomes improve along with IMPACT scores: The average effect of an exit from a school/grade/year cell on the change in the IMPACT score of teachers is about 15-17 points (from a mean of around 280). The effect on students’ math (reading) test score is 0.08 (0.05) standard deviations (SD).
• There is heterogeneity in the effects of turnovers: As expected, the effects are negative on both fronts for school/class/year cells that lost a high-performing teacher, and positive for schools who lost a low-performing teacher. The effects of the former on student achievement are smaller (-0.05 SD) and not statistically significant, while the effects of the latter are larger (0.14-0.21 SD) and significant at the 99% level of confidence.
• High-performers in high-poverty schools are hard to replace: Their exits cause large decreases in IMPACT scores and student achievement (even though the latter do not reach statistical significance due to low power in this cell). In contrast, the overall positive effects on student achievement (and teacher IMPACT scores) are due to the replacement of low-performing teachers in high-poverty schools.

Obviously, this is not an evaluation of the IMPACT program and there would be other considerations and outcomes that might go into such a comprehensive evaluation. But, all else equal, the findings are suggestive of positive impacts on student performance, which might have contributed to the recent improvement of DCPS. Perhaps Mr. Fenty and Ms. Rhee can sleep a little better knowing that what they helped start in the district may have done the students some good.