Syndicate content

evaluation

‘Relevant Reasons’ in Decision-Making (3 of 3)

Heather Lanthorn's picture

This is the third in our series of posts on evidence and decision-making; also posted on Heather’s blog. Here are Part 1 and Part 2
***
In our last post, we wrote about factors – evidence and otherwise – influencing decision-making about development programmes. To do so, we have considered the premise of an agency deciding whether to continue or scale a given programme after piloting it and including an accompanying evaluation commissioned explicitly to inform that decision. This is a potential ‘ideal case’ of evidence-informed decision-making. Yet, the role of evidence in informing decisions is often unclear in practice.

What is clear is that transparent parameters for making decisions about how to allocate resources following a pilot may improve the legitimacy of those decisions. We have started, and continue in this post, to explore whether decision-making deliberations can be shaped ex ante so that, regardless of the outcome, stakeholders feel it was arrived at fairly. Such pre-commitment to the process of deliberation could carve out a specific role for evidence in decision-making. Clarifying the role of evidence would inform what types of questions decision-makers need answered and with what kinds of data, as we discussed here.

Two new rigorous evaluations of technology use in education

Michael Trucano's picture

Look, right there, there it is: Impact! (I think ...)Last week saw a flurry of news reports in response to a single blog post about the well known One Laptop Per Child project. It's dead, proclaimed one news report as a result; it's not dead yet, countered another. Recalling Mark Twain's famous quotation, Wired chimed in to announce that Reports of One Laptop Per Child's death have been greatly exaggerated.

Whatever the status and future of the iconic initiative that has helped bring a few million green and white laptops to students in places like Uruguay, Peru and Rwanda, it is hard to argue that, ten years ago, when the idea was thrown out there, you heard a lot of people asking, ‘Why would you do such a thing?’ Ten years on, however, the idea of providing low cost computing devices like laptops and tablets to students is now (for better and/or for worse, depending on your perspective) part of the mainstream conversation in countries all around the world.

What do we know about the impact and results of initiatives
to provide computing devices to students
in middle and low income countries around the world?

Have Evidence, Will… Um, Erm (2 of 2)

Heather Lanthorn's picture

This is the second in a series of posts with suvojit, initially planned as a series of two but growing to six…

Reminder: The Scenario
In our last post, we set up a scenario that we* have both seen several times: a donor or large implementing agency (our focus, though we think our arguments apply to governmental ministries) commissions an evaluation, with explicit (or implicit) commitments to ‘use’ the evidence generated to drive their own decisions about continuing/scaling/modifying/scrapping a policy/program/project.

And yet. the role of evidence in decision-making of this kind is unclear.

In response, we argued for something akin to Patton’s utilisation-focused evaluation. Such an approach assesses the “quality” or “rigor” of evidence by considering how well it addresses the questions and purposes needed for decision-making with the most appropriate tools and timings to facilitate decision-making in particular political-economic moment, including the capacity of decision-makers to act on evidence.

Have Evidence, Will… Um, Erm?

Heather Lanthorn's picture

Commissioning Evidence

Among those who talk about development & welfare policy/programs/projects, it is tres chic to talk about evidence-informed decision-making (including the evidence on evidence-informed decision-making and the evidence on the evidence on…[insert infinite recursion]).

This concept — formerly best-known as evidence-based policy-making — is contrasted with faith-based or we-thought-really-really-hard-about-this-and-mean-well-based decision-making. It is also contrasted with the (sneaky) strategy of policy-based evidence-making. Using these approaches may lead to not-optimal decision-making, adoption of not-optimal policies and subsequent not-optimal outcomes.

In contrast, proponents of the evidence-informed decision-making approach believe that through their approach, decision-makers are able to make more sound judgments between those policies that will provide the best way forward, those that may not and/or those that should maybe be repealed or revised. This may lead them to make decisions on policies according to these judgments, which, if properly implemented or rolled-back may, in turn, improve development and welfare outcomes. It is also important to bear in mind, however, that it is not evidence alone that drives policymaking. We discuss this idea in more detail in our next post.

Promoting literacy with mobile phones in rural Papua New Guinea

Michael Trucano's picture

hey, my ears are ringing -- might that be the Ministry of Education calling with today's lesson?Last year I spent some time in Papua New Guinea (or PNG, as it is often called), where the World Bank is supporting a number of development projects, and has activities in both the ICT and education sectors. For reasons historical (PNG became an independent nation only in 1975, breaking off from Australia), economic (Australia's is by far PNG's largest export market) and geographical (the PNG capital, Port Moresby, lies about 500 miles from Cairns, across the Coral Sea), Australia provides a large amount of support to the education sector in Papua New Guinea, and I was particularly interested in learning lessons from the experiences of AusAid, the (now former) Australian donor agency.

For those who haven't been there: PNG is a truly fascinating place. It is technically a middle income country because of its great mineral wealth but, according to the Australian government, "Despite positive economic growth rates in recent years, PNG’s social indicators are among the worst in the Asia Pacific. Approximately 85 per cent of PNG’s mainly rural population is poor and an estimated 18 per cent of people are extremely poor. Many lack access to basic services or transport. Poverty, unemployment and poor governance contribute to serious law and order problems."

Among other things, PNG faces vexing (and in some instances, rather unique) circumstances related to remoteness (overland travel is often difficult and communities can be very isolated from each other as a result; air travel is often the only way to get form one place to another: with a landmass approximately that of California, PNG has 562 airports -- more, for example, than China, India or the Philippines!) and language (PNG is considered the most linguistically diverse country in the world, with over 800 (!) languages spoken). The PNG education system faces a wide range of challenges as a result. PNG ranks only 156th on the Human Development Index and has a literacy rate of less than 60%.  As an overview from the Australian government notes,

"These include poor access to schools, low student retention rates and issues in the quality of education. It is often hard for children to go to school, particularly in the rural areas, because of distance from villages to schools, lack of transport, and cost of school fees. There are not enough schools or classrooms to take in all school-aged children, and often the standard of school buildings is very poor. For those children who do go to school, retention rates are low. Teacher quality and lack of required teaching and educational materials are ongoing issues."

[For those who are interested, here is some general background on PNG from the World Bank, and from the part of the Australian Department of Foreign Affairs and Trade that used to be known as AusAid, a short report about World Bank activities to support education in PNG from last year and an overview of the World Bank education project called READ PNG.]

If you believe that innovation often comes about in response to tackling great challenges, sometimes in response to scarcities of various sorts, Papua New Guinea is perhaps one place to put that belief to the test.
 

Given the many great challenges facing PNG's education sector,
its low current capacity to meet these challenges,
and the fact that 'business as usual' is not working,
while at the same time mobile phone use has been growing rapidly across society,
might ICTs, and specifically mobile phones,
offer new opportunities to help meet many long-standing, 'conventional' needs
in perhaps 'unconventional' ways?

A small research project called SMS Story has been exploring answers to this question.

#4 from 2013: Numbers Are Never Enough (especially when dealing with Big Data)

Susan Moeller's picture

Our Top Ten Blog Posts by readership in 2013
This post was originally published on January 8, 2013


The newest trend in Big Data is the personal touch.  When both the New York Times and Fast Company have headlines that trumpet: “Sure, Big Data Is Great. But So Is Intuition.” (The Times) and “Without Human Insight, Big Data Is Just A Bunch Of Numbers.” (Fast Company) you know that a major trend is afoot.

So what’s up?

The claims for what Big Data can do have been extraordinary, witness Andrew McAfee and Erik Brynjolfsson’s seminal article in October in the Harvard Business Review: “Big Data: The Management Revolution,” which began with the showstopper:  “‘You can’t manage what you don’t measure.’”  It’s hard not to feel that Big Data will provide the solutions to everything after that statement.  As the HBR article noted:  “…the recent explosion of digital data is so important. Simply put, because of big data, managers can measure, and hence know, radically more about their businesses, and directly translate that knowledge into improved decision making and performance.”

Ups and Downs in the Struggle for Accountability – Four New Real Time Studies

Duncan Green's picture

OK, we’ve had real time evaluations, we’ve done transparency and accountability initiatives, so why not combine the two? The thoroughly brilliant International Budget Partnership is doing just that, teaming up with academic researchers to follow in real time the ups and downs of four TAIs in Mexico, Brazil, South Africa and Tanzania. Read the case study summaries (only four pages each, with full versions if you want to go deeper), if you can, but below I’ll copy most of the overview blog by IBP research manager Albert van Zyl.

By following the work rather than tidying it all up with a neat but deceitful retrospective evaluation, they record the true messiness of building social contracts between citizens and states: the ups and downs, the almost-giving-up-and-then-winning, the crucial roles of individuals, the importance of scandals and serendipity.

What is a Theory of Change and How Do We Use It?

Duncan Green's picture

I’m planning to write a paper on this, but thought I’d kick off with a blog and pick your brains for references, suggestions etc. Everyone these days (funders, bosses etc) seems to be demanding a Theory of Change (ToC), although when challenged, many have only the haziest notion of what they mean by it. It’s a great opportunity, but also a risk, if ToCs become so debased that they are no more than logframes on steroids. So in internal conversations, blogs etc I’m gradually fleshing out a description of a ToC. When I ran this past some practical evaluation Oxfamers, they helpfully added a reality check – how to have a ToC conversation with an already existing programme, rather than a blank sheet of paper?

But first the blank sheet of paper. If you’re a regular visitor to this blog, you’ll probably recognize some of this, because it builds on the kinds of questions I ask when trying to understand past change episodes, but throws them forward. Once you’ve decided roughly what you want to work on (and that involves a whole separate piece of analysis), I reckon it’s handy to break down a ToC into four phases, captured in the diagram.

Videogames and Learning

Michael Trucano's picture

playing to learn?Not a week goes by where I don't receive an unsolicited email from a company touting the benefits of its new 'educational videogame'. Indeed, just last week I opened my inbox to find two separate emails proclaiming how two different mobile gaming apps were destined to "transform learning!!!" Now, in a lot of the cases, I must confess that I am not always sure why something is an 'educational game', and not just a 'game' (although if I am in a difficult mood, I might offer that in too many instances an 'educational game' is 'a game that really isn't much fun'). That said, there is no denying that videogames are big business around the world. So -- increasingly -- is education. Even most people who fear that potential negative effects of some (or even most) videogames on young people would, at the same time, acknowledge the promise and potential for videogames to offer enriching learning experiences. The history of the introduction of educational technologies is in many ways long on promise and potential, however, and short on actual evidence of how they impact learning in tangible and fundamental ways.

Much is made of the potential for ICTs to be used to promote more personalized learning experiences through the introduction of various types of ICT-enabled assessment systems. For me, it has long seemed like the most powerful real-time learning assessment engines have been found in videogames, where actions (or inactions) are often met with near instantaneous responses, to which the player is then challenged to respond in turn. This feedback loop -- taking an action, being presented with information as a result, having to synthesize and analyze this information and doing something as a result -- might meet some people's definition of 'learning'. A good videogame engages its users so strongly that they are willing to fail, and fail, and fail again, until they learn enough from this failure that they can proceed with the game. Even where educational software is not explicitly labeled as a 'game', designers are increasingly introducing game-like elements (badges, achievement bonuses, scoring systems) as a way to promote user (learner? player?) engagement as part of a process known as 'gamification'.

The use of videogames for educational purposes, or at least in educational contexts, is far from an OECD or U.S. phenomenon. Whether I am visiting a school computer lab after hours in central Russia, an Internet cafe filled with students in Indonesia or standing behind some schoolgirls carrying phones between classes in Tanzania, 'educational' videogames seem to be nearly everywhere. Past posts on the EduTech blog have profiled things like the use of video games on mobile phones to promote literacy in rural India and EVOKE, an online game for students across Africa which the World Bank helped sponsor a few years ago. When I speak with young software entrepreneurs in Nairobi or Accra or Manila, they often talk excitedly about the latest educational game they are developing (for markets local and distant).

Do educational games 'work'?
Are they 'effective'?
And if so: How can they be used in schools?

Questions such as these are of increasing interest to scholars. Given both their potential for learning, and how aggressively videogames are being marketed to many education systems, they should be of increasing interest to educational policymakers as well. Some recent research brings us a little closer to a time when we can answer some of them.

Will Midterm Evaluations Become the Dinosaurs of Development?

Milica Begovic's picture

I argued a few months back that information we get from story-telling is fundamentally different to what we get from polls and surveys. If we can’t predict what’s coming next, then we have to continuously work to understand what has and is happening today. (See: Patterns of voices from the Balkans – working with UNDP)

Methods we’re all used to using (surveys, mid-term evaluations) are ill prepared to do that for us and increasingly act as our blindfolds.

Why stories?

As I started working through the stories we collected, this question has become even stronger.

To give you some background, we started testing whether stories could help us:


Pages