evaluation

Research questions about technology use in education in developing countries

In May last year, key stakeholders joined the World Bank Group in calling for global and more concerted action to address the climate impact of transport while ensuring mobility for everyone. More recently, the Secretary-General’s High-Level Advisory Group on Sustainable Transport noted, in its final recommendations to Ban Ki-Moon, emphasized the need for “coalitions or partnership networks” to “strengthen coherence” for scaling up sustainable transport, as well as establishing monitoring and evaluation frameworks. These issues have been raised at Habitat III, COP22 and at the Global Sustainable Transport Conference in Ashgabat.

As the global community readies itself to move from commitments to implementation, what can transport learn from similar initiatives in other sectors, such as Sustainable Energy for All (SE4All)?

Where have we got to on Theories of Change? Passing fad or paradigm shift?

Theories of change (ToCs) – will the idea stick around and shape future thinking on development, or slide back into the bubbling morass of aid jargon, forgotten and unlamented? Last week some leading ToC-istas at ODI, LSE and The Asia Foundation and a bunch of other organisations spent a whole day taking stock, and the discussion highlighted strengths, weaknesses and some looming decisions.

(Summary, agenda + presentations here)

According to an excellent 2011 overview by Comic Relief, ToCs are an "on-going process of reflection to explore change and how it happens – and what that means for the part we play". They locate a programme or project within a wider analysis of how change comes about, draw on external learning about development, articulate our understanding of change and acknowledge wider systems and actors that influence change.

But the concept remains very fuzzy, partly because (according to a useful survey by Isobel Vogel) ToCs originated from two very different kinds of thought: evaluation (trying to clarify the links between inputs and outcomes) and social action, especially participatory and consciously reflexive approaches.

At the risk of gross generalization, the first group tends to treat ToCs as ‘logframes on steroids’, a useful tool to develop more complete and accurate chains of cause and effect. The second group tend to see the world in terms of complex adaptive systems, and believe the more linear approaches (if we do X then we will achieve Y) are a wild goose chase. These groups (well, actually they’re more of a spectrum) co-exist within organisations, and even between different individuals in country offices.

Social Marketing Master Class: The Importance of Evaluation

Social marketing emerged from the realization that marketing principles can be used not just to sell products but also to "sell" ideas, attitudes and behaviors. The purpose of any social marketing program, therefore, is to change the attitudes or behaviors of a target population-- for the greater social good.

In evaluating social marketing programs, the true test of effectiveness is not the number of flyers distributed or public service announcements aired but how the program impacted the lives of people.

Rebecca Firestone, a social epidemiologist at PSI with area specialties in sexual and reproductive health and non-communicable diseases, walks us through some best practices of social marketing and offers suggestions for improvement in the future. Chief among her suggestions is the need for more and better evaluation of social marketing programs.

Social Marketing Master Class: The Importance of Evaluation

Evaluating the Khan Academy

I just flew in to Cancun this afternoon. The sun’s shining. The sea is blue. The Moon Palace – the site of the negotiations – is a beautiful resort with its own one-kilometer white sandy beach. Jackets and ties are discouraged, and many delegates are wearing the traditional Mexican guayaberas.

The Mexican hosts have done an outstanding job –in diplomacy and logistics – in preparing for this event. So why do the tourists look like they’re having a better time than the delegates?

UNFCCC Executive Secretary Christiana Figueres speaking at the opening ceremony. Photo by IISD Because nobody knows how this will turn out. Everybody feels that somebody else should be putting more on the table, and some are expressing this with great emotion. The excitement of a possible deal last year in Copenhagen is gone. There is no home run, slam dunk or hole in one in the offing. The analogy is now from American Football: It’s about moving the ball patiently down the field with the hope of an eventual touchdown next year, the one after, or five years from now. Above all, don’t drop the ball, or we could lose ground fast.

But this cautious view short-changes what Cancun should achieve. The package of decisions that is being negotiated is highly consequential, and could significantly improve the prospects of a pro-poor climate-friendly future.

So, what would success look like at the end of this 12 day marathon? By the end of this meeting we could have the following:

1. Forests: The first globally agreed REDD+ partnership providing sufficient funding for investments, performance-based payments, and readiness for future carbon market inclusion – thus ensuring that forests are more valued alive than dead.

2. Adaptation: A framework for ensuring the fair and adequate allocation of resources for climate resilient growth, with special attention to the most vulnerable countries, and a process for ensuring lesson learning and technical assistance on this urgent agenda.

‘Relevant Reasons’ in Decision-Making (3 of 3)

This is the third in our series of posts on evidence and decision-making; also posted on Heather’s blog. Here are Part 1 and Part 2
***
In our last post, we wrote about factors – evidence and otherwise – influencing decision-making about development programmes. To do so, we have considered the premise of an agency deciding whether to continue or scale a given programme after piloting it and including an accompanying evaluation commissioned explicitly to inform that decision. This is a potential ‘ideal case’ of evidence-informed decision-making. Yet, the role of evidence in informing decisions is often unclear in practice.

What is clear is that transparent parameters for making decisions about how to allocate resources following a pilot may improve the legitimacy of those decisions. We have started, and continue in this post, to explore whether decision-making deliberations can be shaped ex ante so that, regardless of the outcome, stakeholders feel it was arrived at fairly. Such pre-commitment to the process of deliberation could carve out a specific role for evidence in decision-making. Clarifying the role of evidence would inform what types of questions decision-makers need answered and with what kinds of data, as we discussed here.

Two new rigorous evaluations of technology use in education

Look, right there, there it is: Impact! (I think ...) — _{Look, right there, there it is:

Impact! (I think ...)}

Last week saw a flurry of news reports in response to a single blog post about the well known One Laptop Per Child project. It's dead, proclaimed one news report as a result; it's not dead yet, countered another. Recalling Mark Twain's famous quotation, Wired chimed in to announce that Reports of One Laptop Per Child's death have been greatly exaggerated.

Whatever the status and future of the iconic initiative that has helped bring a few million green and white laptops to students in places like Uruguay, Peru and Rwanda, it is hard to argue that, ten years ago, when the idea was thrown out there, you heard a lot of people asking, ‘Why would you do such a thing?’ Ten years on, however, the idea of providing low cost computing devices like laptops and tablets to students is now (for better and/or for worse, depending on your perspective) part of the mainstream conversation in countries all around the world.

What do we know about the impact and results of initiatives
to provide computing devices to students
in middle and low income countries around the world?

Have Evidence, Will… Um, Erm (2 of 2)

This is the second in a series of posts with suvojit, initially planned as a series of two but growing to six…

Reminder: The Scenario
In our last post, we set up a scenario that we* have both seen several times: a donor or large implementing agency (our focus, though we think our arguments apply to governmental ministries) commissions an evaluation, with explicit (or implicit) commitments to ‘use’ the evidence generated to drive their own decisions about continuing/scaling/modifying/scrapping a policy/program/project.

And yet. the role of evidence in decision-making of this kind is unclear.

In response, we argued for something akin to Patton’s utilisation-focused evaluation. Such an approach assesses the “quality” or “rigor” of evidence by considering how well it addresses the questions and purposes needed for decision-making with the most appropriate tools and timings to facilitate decision-making in particular political-economic moment, including the capacity of decision-makers to act on evidence.

Have Evidence, Will… Um, Erm?

Commissioning Evidence

Among those who talk about development & welfare policy/programs/projects, it is tres chic to talk about evidence-informed decision-making (including the evidence on evidence-informed decision-making and the evidence on the evidence on…[insert infinite recursion]).

This concept — formerly best-known as evidence-based policy-making — is contrasted with faith-based or we-thought-really-really-hard-about-this-and-mean-well-based decision-making. It is also contrasted with the (sneaky) strategy of policy-based evidence-making. Using these approaches may lead to not-optimal decision-making, adoption of not-optimal policies and subsequent not-optimal outcomes.

In contrast, proponents of the evidence-informed decision-making approach believe that through their approach, decision-makers are able to make more sound judgments between those policies that will provide the best way forward, those that may not and/or those that should maybe be repealed or revised. This may lead them to make decisions on policies according to these judgments, which, if properly implemented or rolled-back may, in turn, improve development and welfare outcomes. It is also important to bear in mind, however, that it is not evidence alone that drives policymaking. We discuss this idea in more detail in our next post.

Promoting literacy with mobile phones in rural Papua New Guinea

a longer version of this blog post is available on the MIT Media Lab’s Digital Currency Initiative platform

With Google Trends data showing that searches for the word “blockchain” have exponentially increased, we may be entering the peak of the hype cycle for blockchain and distributed ledger technology.

But here’s the thing: the blockchain is a major breakthrough. That’s because its decentralized approach to verifying changes in important information addresses the centuries-old problem of trust, a social resource that is all too often in short supply, especially amid the current era’s rampant concerns over the security of valuable data. It turns out that fixing that can be a boon for financial inclusion and other basic services delivery, helping to achieve the global objectives laid out in the Sustainable Development Goals (SDGs).

Sorting out hype from reality may depend on how well we identify where institutions that have until now played a role in mediating trust between people are falling short, especially in the key area of money. Deploying the blockchain in those settings to generate secure, decentralized trust could achieve great strides in inclusion and innovation.

What do we mean by decentralized trust? The concept is unfamiliar in part because its converse -- centralized trust – is something that we often take for granted, at least while it’s working. But if we look at the history of transactions since the early barter systems to modern-day digital money exchanges, we can see how different trust protocols for keeping track of our exchanges of value have evolved and how, in each case, centralizing trust within particular institutions has periodically caused problems.

As strategies for dealing with this challenge evolved and as the complexity and frequency of transactions grew, different trust bearers emerged. We went from relying on the memory and discretion of tribal leaders, to central governments issuing currencies in the form of precious metals, to commercial banks acting as trusted intermediaries and issuing their own bank notes, to central banks managing a hybrid system in which sovereign fiat banknotes circulate alongside a debt/credit form of money managed by regulated banks and internal ledgers.

#4 from 2013: Numbers Are Never Enough (especially when dealing with Big Data)

Our Top Ten Blog Posts by readership in 2013
This post was originally published on January 8, 2013

The newest trend in Big Data is the personal touch. When both the New York Times and Fast Company have headlines that trumpet: “Sure, Big Data Is Great. But So Is Intuition.” (The Times) and “Without Human Insight, Big Data Is Just A Bunch Of Numbers.” (Fast Company) you know that a major trend is afoot.

So what’s up?

The claims for what Big Data can do have been extraordinary, witness Andrew McAfee and Erik Brynjolfsson’s seminal article in October in the Harvard Business Review: “Big Data: The Management Revolution,” which began with the showstopper: “‘You can’t manage what you don’t measure.’” It’s hard not to feel that Big Data will provide the solutions to everything after that statement. As the HBR article noted: “…the recent explosion of digital data is so important. Simply put, because of big data, managers can measure, and hence know, radically more about their businesses, and directly translate that knowledge into improved decision making and performance.”

evaluation

Pages

follow us

newsletter