With the World Bank’s assistance, many governments are seeking to expand or improve early childhood education programs. The Strategic Impact Evaluation Fund is supporting a number of evaluations that will help us estimate the benefits of these investments, but we shouldn’t just care about effectiveness. In a world of resource constraints, we also need to tell countries about the cost-effectiveness of these investments. To do this, we need to know the costs of the programs we’re evaluating. Unfortunately, organizations often don’t collect and regularly report intervention specific cost data.
Recently, I helped a non-governmental organization figure out how to do this for a program that provides an extra-year of preschool in a rural district of Bangladesh. The staff I worked with initially assumed that this would be a tedious accounting exercise. However, it quickly became apparent to everyone that collecting costs was really an investigation, one that takes you deep into the nuts-and-bolts of program design and implementation. And in this particular investigation, our initial findings were surprising: if we had relied on the initial “budget information” the program was keeping, we would have underestimated total costs of the program by a factor of five.
When I first arrived, the program team gave me their district level budget (let’s call this Budget 1). But the budget wasn’t specific to the program offering the extra year of preschool; it included some of the costs for primary school interventions being carried out concurrently in the same region. More importantly, many people and materials critical for the program did not show up in this budget, like management and overhead costs, travel and accommodation costs for project managers, learning materials procurement, computers and motorbikes, office and utility costs, and community inputs (such as additional teacher stipends and parent volunteers ). Some of these costs listed above were counted in the organization’s country office accounting and budget data. Others, like community costs, which would be resources or time and effort put into the program by community members, weren’t recorded at all since the organization didn’t pay for them.
So I knew Budget 1 would not give me an accurate estimate of program cost. I explained to the finance staff and program manager that it was necessary to list and value all inputs required to make the extra year of preschool happen. This process is called the “ingredients method.” This not only involves collecting all the budget and financial data, but also disaggregating that data by adding often absent variables like unit prices, quantities, and the date of expenditure. This type of information would be important to know when scaling up the program later, as prices of products purchased in bulk and in different locations might differ.
Perhaps the most challenging variable is determining how much of a shared resource, such as labor, should be allocated to the intervention. For example, a senior officer split duties between the primary and extra-year preschool programs, so only 50 percent of his salary was included in the cost model. This level of disaggregation often requires in-person interviews and possibly real-time data collection as the program is being implemented, as people often find it difficult to recall how much time they have spent on a task. After explaining the type of disaggregated data necessary to do cost-analysis, the finance team spent a week compiling a new dataset. The result was a budget five times larger than the original district budget! Let’s call this Budget 2.
But I knew that Budget 2 wasn’t going to be our final budget either. Learning materials were still listed as a single line item, and the figure ($5000) just seemed too round and neat to me to be true. After I obtained a materials list with information about each item’s price and quantity, the materials cost doubled (Budget 3). We then disaggregated further by inquiring how often these materials were purchased, since not all materials are purchased each year, and since some other program costs were startup costs (only incurred in the first year), the cost decreased (Budget 4).
Here’s a brief visualization of how costs changed as I continued interviewing program staff about costs and adjusting the model as we disaggregated the data further:
So what’s the final budget? I don’t know. This iterative disaggregation exercise didn’t tell me about actual expenditures, as the program is still running. It also didn’t tell me much about the value of community costs. However, the exercise did help me identify for the program team what data to collect regularly during program implementation (which included data on community inputs). At the end of the school year (in December), we’ll be able to finalize an estimate of all expenditures required to make the program happen for this year, which we’ll then be able to match with the evaluation’s estimate of program impact.
That way, when the World Bank discusses the program with the government, it will be possible to present both a robust and rigorous measure of the benefits of the program and an accurate estimate of the costs.
Join the Conversation