Syndicate content

Measuring Public Opinion in Challenging Contexts

Anne-Katrin Arnold's picture

As we have discussed in other blog posts, public opinion is particularly important in countries with weak institutions of governance and accountability. Especially in fragile and conflict states, it can lend legitimacy to the government, help creating a national identity, and support governance reform. Unfortunately, public opinion is particularly hard to measure in those societies where it could be most important.

At the conference on Transnational Connections: Challenges and Opportunities for Public Opinion  in Segovia, Spain, in mid March, I participated in a panel on "Challenges of studying public opinion in challenging contexts." The first presenter, Monroe Price from the University of Pennsylvania, opened the panel with two fundamental questions: does public opinion take the same forms in challenging environments as it does elsewhere, and can we measure it with the same instruments? Mahmood Enayat from the Oxford Internet Institute discussed his experience with opinion surveys in Iran, focusing on methodology and suggesting online and phone surveys as appropriate methods. Ibrahim Al Marashi from IE University (Spain) reported on opinion research in Iraq and problems of interviewers to get at people's real opinions in societies where politeness and hospitality are defining aspects of culture. Julia Shimko, also IE University, talked about corruption surveys that she conducted on behalf of the World Bank and EBRD in former soviet countries, where standardized questionnaires were not successful in providing a picture of the reality of corruption.

Even in open societies it is rather difficult to measure public opinion since we're not so sure after all what public opinion actually is, or rather how it is manifested. This problem is even more dramatic in challenging political environments. Conventional methods such as surveys may not be suitable. The lack of polling institutions and the lack of official regime support for opinion polling may lead organizations to turn to online surveys. Here, of course, we have a huge problem with sampling because Internet access is not prevalent in the developing world and respondents would most often constitute a part of the elite, not a part of the population. Phone surveys, landlines or cell phones, may also be problematic because phones are registered - several countries plan to register cell phones - and can therefore be monitored.

Going back to Monroe Price's question that he posed at the conference: does public opinion take the same forms in challenging environments than in those areas where polling has been practiced for almost a century? Does it build the same way? Since we can't answer these questions, I suggest that small group deliberation may provide a feasible approach to study how public opinion is formed and which solutions it comes up with.

The Indian constitution has a provision to hold public hearings, Gram Sabhas and Gram Panchayats, to involve citizens in policy making and to try overcoming gender and caste differences in the way public opinion is formed. These meetings work somewhat similar to what we know as deliberative polling: an issue is introduced, information about it is provided, and a moderated discussion follows. It's not always successful at including all voices and producing policy relevant results, but it does give a better approximation of public opinion than, say, an online survey in rural India would do. China is working with deliberative polls at the local level where citizen meetings are part of determining budget priorities. We know similar approaches from participatory budgeting in Porto Allegre.

The advantage of these small group deliberations is that they approximate the ideal democratic situation of deliberation better than any opinion poll. People know what the issues are about, they receive information, discuss with their peers, and reach a consensus on solutions. True, power plays a role even in these settings and is likely to distort the outcome. Also true, research shows us that sometimes deliberation does not produce consensus, but exacerbates polarized opinion. And true that the outcome of small group deliberation can be very different that we would see on a larger national level. However, the opinion formation process in small deliberative forums is more ideal than in opinion polls and may give us the best possible idea about how opinion is formed and what it would be if people only had the chance for open, equal, and well informed discussion. This is true for any environment, and it may be the only feasible way to gauge public opinion in closed societies.

Picture: Flickr user magnusfranklin


Indeed these are interesting and timely issues to discuss. Online/mobile Surveys I am not sure, in how far online or mobile surveys should be better suited than standard surveys. Indeed one of their main advantages is anonymity, so that persons might be more inclined to answer to "a robot" rather than an interviewer. On the other side, the methodological issues you raise are all true (sample, representativity, etc), but I would also like to stress two more: - online or mobile surveys are prone to strong self-selectivity effects, meaning that individuals, who have a strong opinion about a specific matter are more prone to enter the survey at all, because they want to make themselves heard. That means that there is a high chance that results oversample somewhat "extreme" opinions. - the majority of online or mobile surveys in both deeloping and developed countries are usually not free in that regard that the respondent has to pay connection costs to answer. That adds to self-selection. CATI surveys Regarding phone surveys, there are ways to obtain a random selection, i.e. through RLD (random last digit) or RDD (random dial digits) which are well field tested. Rather than drawing from a phone book (and running the risk of selectivities of people, who are actually listed!) only a few parameters are needed (overall range of valid numbers, i.e. 914-11111111 to 914-599999) from which numbers are generated and automatically are called. However, this requires quite a lot of senior CATI-IT knowledge to set up. Standard Questionnaires I would like to know more about the issues run into the corruption survey. If the "standardized questionnaires were not successful in providing a picture of the reality of corruption", for me the question would be "why?". I do not think that the fault lies here in using standard questionnaires in itself (online surveys won't help too much, either, because these are essentially standardized surveys that are administered by a computer). I would suspect that the wording is at fault here ("unofficial payments/gifts") or other issues such as identity of interviewers, sample design, etc. Corruption is a highly localized and sensitive issue, so one needs to make sure that the right words are used - in this case it is extremely helpful to work with local partners and to run qualitative interviews to get a feeling of how corruption is described and what questions can be asked and which not. For example, I think that the TI survey on Corruption does a good job in getting data in, so I do not think using standard surveys per se are at fault here. It depends more if the questionnaire is able to describe a situation, which is understandable and relevant to a number of local cultures. As a last comment: I do not think that "public opinion surveys" can be described as a monolithic approach. Just looking at the participants and their studies I would think that they all tried to research different issues in different countries, so in the end I would think that if they succeeded with one study in one country, the question remains if the same instrument/approach will work in another. So, I guess my answers to these questions would be: "does public opinion take the same forms in challenging environments as it does elsewhere?" -- yes, in that regard that measuring "public opinion" is always challenging. "can we measure it with the same instruments?" - not always, but sometimes

Add new comment