Syndicate content

An underappreciated benefit of experiments: convincing politicians when their pet projects don’t work

David McKenzie's picture

New York City has suspended payments in a pay-for-performance program for teachers after an experiment found the program had not worked. From the New York Post:

Another attempt by the city to improve student performance through cash payments has failed, much to the surprise of Mayor Bloomberg.

"I would have thought it would have had a bigger effect," the mayor said, responding to findings that millions of dollars in bonuses paid to teachers over the last three years to boost student test scores didn't do a thing to upgrade their underperforming schools.

The pilot program paid annual performance bonuses of $1,500-3,000, with more than $56 million spent over the first two years. The experiment found no impacts on student achievement at any grade level.

The study was conducted by the RAND corporation. Some key points:

·         427 high-needs schools were randomly allocated to treatment and control groups

·         Participation was voluntary – 55% of staff at treatment schools had to agree to be in the program for the program to be implemented. 205 schools participated in the first year.

·         The comprehensive (312 page) report uses a mixture of experimental analysis via survey and administrative data, detailed interviews and case studies.

·         Point estimates were actually for lower test scores in the treatment schools, but the difference isn’t significant.

·         Schools decided how performance bonuses would get distributed among teachers – most opted for equal shares for all teachers.

·         Many winners reported that, after taxes, the bonus seemed insignificant. Buy-in for the program was also limited, with most teachers feeling it relied too much on test scores.

·         Another hypothesis put forward was that the existing accountability pressures and incentives were already inducing staff to work hard, so that an additional (small) financial bonus had no further effect.



The NY post also reports that: Last year, the city pulled the plug on another $50 million experiment, called "conditional cash transfers," which was modeled on a successful program in Mexico that pays parents to take their kids to school, for health checkups and to engage in other behavior deemed beneficial to society. The city's version included payments when students passed state exams. Despite the cash, there was little difference in outcomes between the paid students and a control group that didn't get a cent.

The mayor argued there's only one way to know if novel approaches are going to work -- and that's to try them. "I think those things we should be more proud of -- the fact that we have the courage to sit there and say we thought it was a good idea, didn't work and we're stopping it," he said. "We're not going to waste the public's money."

Comments

Submitted by Anonymous on
Are there working papers for these failed projects? These seem like important replications, which should be discussed in the literature, rather than buried. The failure of conditional cash transfers in the US seems an especially important replication.

Roland Fryer has a forthcoming paper in the QJE which discusses these educational experiments in the USA. http://www.economics.harvard.edu/faculty/fryer/files/Incentives_5_9_2011.pdf

I agree that its terrific when programs are tested and canceled when they don't work. But it's also important to note that the Bloomberg Administration canceled the OpportunityNYC (a CCT) program after MDRCs RCT showed found significant positive effects. Apparently the effects weren't large enough to provide the political cover necessary. Background and links in the second half of this post: http://bit.ly/9SAw7b