An underappreciated benefit of experiments: convincing politicians when their pet projects don’t work

New York City has suspended payments in a pay-for-performance program for teachers after an experiment found the program had not worked. From the New York Post:

Another attempt by the city to improve student performance through cash payments has failed, much to the surprise of Mayor Bloomberg.

"I would have thought it would have had a bigger effect," the mayor said, responding to findings that millions of dollars in bonuses paid to teachers over the last three years to boost student test scores didn't do a thing to upgrade their underperforming schools.

The pilot program paid annual performance bonuses of $1,500-3,000, with more than $56 million spent over the first two years. The experiment found no impacts on student achievement at any grade level.

The study was conducted by the RAND corporation. Some key points:

·         427 high-needs schools were randomly allocated to treatment and control groups

·         Participation was voluntary – 55% of staff at treatment schools had to agree to be in the program for the program to be implemented. 205 schools participated in the first year.

·         The comprehensive (312 page) report uses a mixture of experimental analysis via survey and administrative data, detailed interviews and case studies.

·         Point estimates were actually for lower test scores in the treatment schools, but the difference isn’t significant.

·         Schools decided how performance bonuses would get distributed among teachers – most opted for equal shares for all teachers.

·         Many winners reported that, after taxes, the bonus seemed insignificant. Buy-in for the program was also limited, with most teachers feeling it relied too much on test scores.

·         Another hypothesis put forward was that the existing accountability pressures and incentives were already inducing staff to work hard, so that an additional (small) financial bonus had no further effect.

The NY post also reports that: Last year, the city pulled the plug on another $50 million experiment, called "conditional cash transfers," which was modeled on a successful program in Mexico that pays parents to take their kids to school, for health checkups and to engage in other behavior deemed beneficial to society. The city's version included payments when students passed state exams. Despite the cash, there was little difference in outcomes between the paid students and a control group that didn't get a cent.

The mayor argued there's only one way to know if novel approaches are going to work -- and that's to try them. "I think those things we should be more proud of -- the fact that we have the courage to sit there and say we thought it was a good idea, didn't work and we're stopping it," he said. "We're not going to waste the public's money."