Syndicate content

Tell us – is there a missing market for collaboration on surveys between WB staff, researchers and students?

David McKenzie's picture

The thought has occurred to me that there are more people than ever doing surveys of various sorts in developing countries, and many graduate students, young faculty, and other researchers who would love the opportunity to cheaply add questions to a survey. I therefore wonder whether there is a missed opportunity for the two sides to get together. Let me explain what I’m thinking of, and then let us know whether you think this is really an issue or not.

There are now a wide range of surveys being taken by researchers in developing countries. The fact that these surveys are researcher-designed and driven potentially offers a lot more scope for customization than in the typical surveys collected by national statistics institutes.  But it seems to me that many of these surveys are underused – surveys collected for impact evaluations are often quite specific and may not be used for anything other than the impact evaluation for which it was designed, while even large multi-purpose cross-sectional surveys like the World Bank’s Living Standards Measurement Surveys seem to be used less and less for research given the difficulties of asking many causal questions with a cross-section.

What I am thinking of here is the possibility of adding a handful of questions or at most a very short module to a survey being undertaken – this takes account of the fact that time and the risk of respondent fatigue is often a big constraint in these surveys, so adding more than say 2-10 minutes of questions to most surveys could be hard. My questions are then whether there are lots of people who potentially benefit from adding a small number of questions to a larger survey (the supply side) and whether there are people who might be willing to consider doing this for surveys they are working on (the demand side).

The Supply Side (of more Survey questions)

It seems likely that there are lots of graduate students out there (and perhaps even undergraduates doing honors theses) who would love to add some questions to a larger survey to test a particular theory, model or insight they have. At some schools this can be done by piggybacking on an advisor’s survey, but for students at many schools this won’t be possible. What is less clear to me is whether there are many types of questions people might be interested in adding which are short enough to not be much of a burden on an existing survey. Some examples which occur to me are:

·         Adding say questions on some parameter of a utility function – e.g. if risk aversion or discount rates, or ambiguity aversion is not going to be asked, adding this to the questionnaire might allow the data to be used to model particular behaviors. Gharad Bryan’s job market paper from last year is an example, which re-uses data from two randomized experiments which happened to collect ambiguity aversion and risk aversion to test the role of ambiguity aversion in insurance decisions.

·         Adding a few questions on some economic behavior that is not typically linked with another – e.g. asking some questions about household bargaining in a firm survey.

·         Testing how to best measure something by randomly varying a couple of questions across different survey units.

There may also be demand for this sort of addition from faculty members and World Bank staff. Faculty might want to add a few questions to a survey to either examine whether their results from one country apply in another, or perhaps to gauge demand for a new intervention they are planning. World Bank staff may need some descriptive data on something that is not typically asked in surveys – e.g. when working on the 2007 World Development Report, we paid to add 10 questions to nationally representative surveys taking place in 7 countries in order to get data on who makes decisions for youth on things like marriage, schooling, and work – something we couldn’t find in other surveys.

The Demand Side (or would anyone agree to add more questions to their survey)?

So it seems plausible that there is plenty of potential supply of additional questions for surveys. But it may be harder to think that there is demand or willingness to add them. But I think there are several reasons why it may not be impossible:

·         The altruistic or public good argument: some researchers or operational staff doing a survey might be willing to add a very small number of questions out of desire to help others; others may have sold their survey to funders as a public good, and this argument would be more credible if they actually offered space on their survey to others – there could easily be points for doing this on a National Science Foundation or World Bank grant application.

·         Quid pro quo? What do grad students have to offer in exchange for getting their questions put on a survey? Free labor might be one answer. This might come in the form of an additional set of eyes and ears on the ground to help ensure the survey runs smoothly and to collect some qualitative information to supplement the survey (e.g. the student might be able to get funding from their university or self-finance the costs of accompanying the survey in the field); or it might come in the form of the offer of some time spent cleaning the data or contributing to a summary report on the data.

·         Co-authorship? This might be less attractive for people writing a dissertation, but for others, if someone comes with a good idea for some questions to add value to the survey, there may be an opportunity for real collaboration.

Is the current equilibrium efficient? If not, what should be done about it?

So maybe there is both a potential supply and demand for new questions. Maybe the level of collaboration we see is efficient, with people with good ideas or something to offer having no trouble finding people doing surveys which they can add their questions to. This might indeed be the case for more senior researchers and for graduate students at institutions with a lot of faculty doing survey work. But it seems less likely for researchers in developing countries, or for students outside a select few schools, who may have more difficulty making these contacts. So it seems possible there are missed opportunities.

What do you think? Let us know in the comments and take the poll  (scroll down on the left of the Development Impact webpage, it is a one question poll) so we can gauge interest. If there is a missing market, what should/could be done about it? I’m happy to have a go being matchmaker (including with a survey of my own) through this blog, but don’t want to get deluged by hundreds of people wanting to add questions if there is no one willing to consider adding them. Any other ideas?

Comments

Submitted by Suvojit on
There does seem to be a gap at the moment. While working in both India and Ghana, I often enountered researchers, who for various reasons did not have the resources to run their own household surveys - possibly meaning that many great ideas were never tested and research questions never answered. And whenever we talked about the panel data that Yale and ISSER are collecting in Ghana, researchers usually got really excited and were interested in finding ways to collaborate. Some of the RCTs I worked on were happy to make their questionnaires and data public. Keeping this in mind, trying to align at least some parts of a specific study questionnaire to the national surveys (by asking similar questions; using similar options and codes for answers) seemed like a great way to create more public goods in the survey-world

As someone at a non-R1 institution this would be great! I think the lowest hanging fruit would be for panel data where a baseline has already been collected. If I can look through a first round survey instrument, I can get a sense of what might be interesting to me in round n+1.

Submitted by Gustavo Andrade on
It would be great. Fortunately, I had this oportunity to research about informal market with the support of the World Bank and local Government, because I could never do this research by myself. But this opportunity is not open to everybody, I'm very luck to have it. If more people could have this oportunity they will be thankfull to work in partnership with the WB.

Submitted by Johan on
These are interesting thoughts David. While the market you describe might not be entirely missing, it is not operating efficiently. Market failure here is driven to some extent by imperfect information and high transaction costs. Both of these could be corrected, but how and by whom? Where does one go to exchange goods and services in this market? Currently successful transactions between Bank staff and external researchers and students hinge primarily on information shared through professional relationships. But this information is, for the most part, not public knowledge. Well let me qualify that. Information on planned surveys (i.e., vehicles for adding questions) is rarely public knowledge, but since April 2011 information and data collected from a wide range of surveys financed and/or coordinated by the World Bank are being made publically available via microdata.worldbank.org. The Bank’s Microdata Library will not help researchers and students with demand to add specific questions to planned surveys, rather it supplies data from surveys that have already been collected, many of which have been used to produce Bank research outputs (the microdata are linked to citations index). Developed by my colleague Olivier Dupriez, this on-line microdata library allows users to search for keywords down to the variable level (currently the library already contains over 600 surveys and almost 375,000 variables) and in a matter of seconds provides a list with surveys and variables/questions that meet the search criteria. The Microdata Library has several other useful functionalities and features. I would encourage you to try it out and to deposit more datasets! And share any ideas you have on how to measure impact of providing this public good? A similar on-line market place that facilitates the exchange of information on planned surveys is missing. Early results from your poll suggest there is latent demand and supply, so testing (and measuring the impact of) a user-friendly on-line market place to reduce transaction costs might make sense.

Submitted by Simone van Dijk on
First I would like to say, a great initiative!As Phd student working on an impact evaluation tool for post-disaster shelters, survey and data supply are the basis and thus strongly needed. And with that I am not the only PhD student, ther are many more in the same position. For me, collaborating with the Netherlands Red Cross, gives me the opportunity to run some household surveys in different areas. Nevertheless, the more data the better. This puts me on the demand as well as the supply side. One of the things that might be a problem is the quality control and the manner of data collection. On the supply side, when adding questions to someone's survey, how would one know how data is gathered?Is there a match between methodology through which the is data gathered through added questions to a survey, and does this match with the data already gathered for example? Something to think about..... As for trial or testing of this idea....I would love to cooperate. Surveys in Indonesia and Colombia are planned from September until June 2012.

Hi Simone, I am connected to some research being done in Merapi, Indonesia with the University of Jakarta on communities that were rebuilt after the tsunami. Surveys are being collected this fall. Also, I am connected to Domes for the World - dftw.org - and there former CEO is now doing more research in this area and is a good friend. I speak spanish fluently and have my own research projects in Mexico. If you are interested in collaborating at all - or contacting my friends in Indonesia or DFTW, let me know jeffswin@gmail.com

David, As a graduate student of development myself, let me tell you - I would love to be able to add questions to a world bank survey. I can see value in doing this - particularly with new methodologies that the WB researchers are interested in exploring. I'm doing a survey in Mexico in villages around a rural telecenter about the impact of using the internet on social life. We are trying to operationalize sen's freedoms methodology into survey methods. Does internet use expand social, political, economic freedoms? I think that if a WB researcher had a particular interest, like how to operationalize Sen's freedoms for impact assessments of ICT4D projects, for example, then it would make sense to add these questions as a trial. However, adding too many questions always brings up survey fatigue. To decide which questions are included, the WB could either hold a competition for best survey questions that operationalize a certain theory, or they could simply solicit questions from grad students at universities where the bank has strong ties. Jeffrey Swindle @jeffswin