Tuesday, September 27, 2005

Polling polls, uncertainty and the War on Science

The Mystery Pollster reviews Gallup's Poll on Polls:
  • 73% of a sample of American adults say the nation would be better off if our leaders "paid more attention to public opinion," and 22% say we would be worse off.
  • 61% on the very next question say the nation would be better off if our leaders paid more attention to "polls," and 33% say we would be worse off.
MP notes that this means "polls" have a more pejorative sense than public opinion, partly because of this result:
"As you may know, most national polls are typically based on a sample of 1,000 adults.  Do you think a sample of this size accurately reflects the views of the nation's population or not?"  30% say yes, 68% say no. 

As a statistician, I can tell you that – barring sampling error – a sample of 1000 people gives tremendous statistical power. To shrink the margin of error by a constant amount, you need ever larger increases in sample size. It's remarkable, but a simple, random sample of 1,000 people is a pretty good indicator of what a country of 350,000,000 think (not perfect, but surprisingly good). The other surprising result is that a sample of 2,000 people isn't that much better than a sample of 1,000.

This reflects a key point about the human mind: people can't grasp uncertainty on an intuitive level. It takes a course in stats plus a year or two of practical experience before that worldview makes sense.

Who cares?

Well, the idea of seeing a population as something represented by a measure of location (an average) and a measure of dispersal (standard error, variance, even the range) is fundamental to evolutionary biology. Humans aren't just defined (contra Quetelet) by being 6 feet tall. We're 6 feet tall plus or minus some amount. Evolution happens when part of that range of variation is more successful at leaving offspring than another part. The Napoleonic wars are a classic example, with some historians claiming that the draft of tall men into Napoleon's army shortened subsequent generations of Europeans. A sexual preference for tall mates would tend to increase average heights.

Darwin's deep insight, a point he spends tremendous effort on in the Origin, is that species and "varieties" (subspecies, breeds or races) are not fundamentally different. Species represent fuzzy clusters of organisms linked by common ancestry, a history of random changes and non-random selection. A variety becomes a species as it becomes more different from it's ancestor and the other descendants of that ancestor. There's no clear boundary, just a statistical process of deviation from a common origin.

When people get that idea on some deep level, they see things differently. Creationism looks childish and trivial. Claims about the superiority of humans look absurd on their face. All life shares more than it differs by, heritage of a common ancestor.

I thought about this as I was pushing to finish up The Republican War on Science by Chris Mooney. It's an excellent book, and a fuller review will come soon (I hoped to have it up weeks ago, but I keep getting delayed). The thread connecting many of the abuses of science Mooney catalogs is the way that industry, religious groups and other interested parties on the Republican side tend to emphasize the existence of any uncertainty about anything as a strike against any consensus at all.

Scientists are accustomed to uncertainty. We deal in stats, we know that our experiments are never perfect, that humans mess up and that experiments can never evaluate every possible alternate hypothesis. We're OK with that because we develop mental constructs to help confront that uncertainty. We look at the commonalities among studies, look for patterns in the disagreements, formulate new hypotheses, and test them. We know that knowledge is contingent, but we can move past that.

The public at large lacks those coping mechanisms. They are mental constructs which we develop and hone in college science classes, but not before. Until college, students hear tales of scientific discovery as established fact, of the brilliant experiment which answers a question so elegantly that no questions remain. Science is presented as an encyclopedia, rather than as a human process subject to all the flaws that humans are subject to. Its strength is the fact that it incorporates internal checks and constant scrutiny. We may not be able to definitively identify the truth, but we are able to nail the falsehoods, and failing to disprove something starts to look like proving it to a scientist.

Does disagreement exist in principle? Sure. Is there variation around the consensus position? Sure. But the consensus exists because all the evidence points in the same direction.

The fact that people are so hostile to the fairly large sample size of 1,000 in polling samples is a symptom if the general failure to discuss uncertainty and random error with the public. It's a failure that can be remedied, and should be.

Technorati Tags: , ,