The first time a World Bank education team tried classroom observations in Brazil, it nearly provoked a state-wide teachers’ strike. It was October 2009 in the northeast state of Pernambuco and two members of the team, Barbara Bruns and Madalena Dos Santos, had handed out stopwatches to school supervisors newly trained in using the Stallings “classroom snapshot” method to measure teacher activities.
Two days later, the stopwatches were on the front page of Pernambuco’s leading newspaper: the teachers’ union called for a state-wide strike to protest an evaluation tool they dubbed the “Stalin method.”
“I thought the grant money we had used to train observers was down the drain,” recalled Bruns, a World Bank retiree now a visiting Fellow at the Center for Global Development. “But the governor, Eduardo Campos, was unfazed. He publicly declared: ‘No one is going to stop me and my secretariat from going into public schools to figure out how to make them better.’ The union backed down and the fieldwork went ahead.”
In an interview, Bruns, who along with the World Bank’s Leandro Costa recently completed a SIEF-supported impact evaluation in Brazil’s Ceara state using Stallings data, talked about what the Stallings method is, why it’s useful for education research, and how 15 seconds—the amount of time needed to capture a single classroom interaction— is actually more time than you think.
How did you become interested in the method?
We wanted to do an impact evaluation of Pernambuco’s school bonus program to gain a deeper understanding of how a pay incentive like a school bonus affects teacher behavior. A couple of high profile evaluations had shown that school bonuses increased student learning, but they couldn’t explain what teachers actually did to achieve this. We looked for a simple classroom observation tool that might help.
How does the Stallings method work?
Observers do a 15-second scan of the classroom and then code exactly what they see. Fifteen seconds is enough to capture a single moment of teacher-student interaction and identify four things: whether the teacher is engaged in instruction; what pedagogical technique the teacher is using; what learning materials are being used; and what share of children are engaged with the teacher versus daydreaming or chatting, which is called “off-task”.
The method calls for 10 evenly-spaced observations in each class. For example, every three minutes in a 30-minute class or every five minutes in a 50-minute class. The fact that observations are so short allows users to code it precisely: at a single moment, a teacher can only be doing one thing.
Stallings can’t capture a teacher’s content mastery or whether she or he is providing emotional support for students, but the variables it does capture have been correlated with classroom level learning gains. There are two key advantages. First, Stallings data are comparable across classrooms, no matter what subject, grade, language or country the teacher is working in, so teacher practice can be compared across different classrooms, schools, cities and countries. Second, the simplicity of the method means that observers don’t need to be highly educated to code reliably.
Why did you want to create a digital version of the technique?
We used to code on paper grids, which needed to be shipped, scanned, and digitized. We must have shipped tens of thousands of pages to teams in the field and then back to consulting firms hired to digitize the data. It was time-consuming and expensive. The tablet-based instrument is simple to use and all the analysis happens in one second: you upload the observations to a server and it generates an excel spread sheet with your results.
So how does this data lead to change?
We worked in Brazil with Rio municipality, where Claudia Costin, who later served as Senior Director of Education at the World Bank, was Secretary. Stallings data showed that teachers’ classroom practice was very traditional and students weren’t engaged. So she introduced a classroom practice dimension into the teacher hiring process; after teachers passed the content tests, they had to deliver a sample lesson. She also discussed the Stallings results with supervisors and school directors looking for ideas to improve instruction and keep students engaged. A second round of observations one year later showed clear improvements.
In Ceara, Leandro and I designed a special bulletin to give schools feedback on their Stallings observation results, After one year, our randomized evaluation found that the observation feedback plus coaching improved teacher practice and raised student learning in math and Portuguese on both a state and national test. This is exciting evidence that classroom observation feedback can help individual teachers.
Education was the subject of this year’s World Development Report and it seems like there are big challenges to getting education right.
The report’s focus on learning is moving us in the right direction. The politics of reform as relates to teachers and the policies for their preparation, recruitment and motivation, are difficult, but not insurmountable. I have had the opportunity to work closely with some incredible education leaders in Latin America and the Caribbean who were truly committed to making education systems better. This showed me first-hand that education systems really can improve. But being able to look into classrooms to see if reforms translate into better teacher practice is essential for getting it right, and the Stallings instrument has proven useful for that.
Find out more about World Bank Education on our website and on Twitter.
Read about SIEF.
Join the Conversation