Let the People be the Judge!

It was in Manila last week where I came across a banner headline on a major broadsheet that read “The people, not surveys, should judge (the president’s) performance."  I was confused.  Aren't people’s attitudes, opinions, and intentions precisely what surveys seek to measure?  Aren’t surveys, in fact, meant to reflect the will and preferences of the people?

When surveys are done well and conscientiously, they provide valuable information from which we can derive knowledge helpful toward understanding people's opinions, especially on matters of public interest.  Applying public opinion research techniques can also aid in improving the quality of democratic governance, particularly in coming to more informed decisions that more closely reflect citizen preferences (e.g., James S. Fishkin’s chapter in Governance Reform under Real-World Conditions).

I then thought about some reasons why people distrust surveys.  For one thing, perhaps the most obvious reason, they aren’t always accurate despite best efforts.  Recent history is chock-full of examples of surveys getting it wrong.  A second criticism I’ve often come across is that many surveys are done badly.  Technical issues abound, such as the lack of accounting for sampling error, question wording, and question order effects, among others.  Third, some surveys are wolves in sheep’s clothing – they aren’t really surveys!  There are those who do damage to the general reputation of public opinion research via questionable practices such as  push polling, where questions are asked with the nefarious intent of influencing respondents, not eliciting their opinions.  For example, a person might be asked “Did you know that candidate “X” has a long history of doing bad thing “Y”? 

Another common problem is the way in which news reports that include survey results often lack or de-emphasize vital information crucial for making meaningful inferences.  For instance, overlapping margins of error denote that differences that may look rather large are actually not statistically significant.  For example, if one policy is found to be preferred over another by six percentage points, but the margin of error is plus or minus four, there is no statistical difference between the two policies.  They are statistically tied, and any statement implying that the public prefers one policy over the other is simply wrong.

Surveys can be done well and many of them are.  See, for example, the excellent work done by the Pew Center, Globalbarometers, WorldPublicOpinion.org, and the National Annenberg Election Survey.  But in general, a healthy skepticism should be encouraged.  Reporters who use surveys in their stories must work with the public opinion research community to come up with a set of global good practices.  The news media, especially, should be discouraged from using straw polls as marketing gimmicks.  Those who practice push polling should be denounced and discredited.  And most importantly, policy makers should have at least one person in their staffs with the technical know-how to vet survey findings. 

The people, and the surveys that reflect their preferences should indeed be used as an input toward improving decision making in governance.  But surveys, even when done well, are mere snapshots, always clouded by some level of distortion.  That said, the quality of governance and citizen participation can be enhanced by the use of surveys insofar as we’re clear about what the images are telling us, and what they’re not.

Photo credit: Flickr user Ahron de Leeuw