Monday, January 30, 2006

The scientific method

The hack ID website I discussed yesterday has posted the results of their hack survey, but like true hacks, failed to understand what they discovered:
A similar yet rarely acknowledged problem is when the respondents in a letter poll correspond with each other, coordinating their replies.

Unfortunately, some ID critics didn’t like the scientific method to be applied to themselves. Within 27 minutes, one of the respondents, Wesley Elsberry of The Austringer, had posted the contents of my letter, advising others to reply by choosing “G”. And within hours, other blogs had followed, including the highly popular Pharyngula. As another respondent, Tara Smith, said, “If you received [a mail], check out their comments before sending your answer back.” Predictably, all of the respondents who replied either chose “G” [none of the above] or refused to participate in the survey (as it was of course their right to do, the survey being voluntary).
Apparently this was a closed book test, but no one was told about that. And it was people's right not to reply, but not to choose option G. Also, respondents were assured anonymity, but there he is, identifying respondents.

Now, 36 people (his sample size) is fairly small for something like this. If the results had been scattered among the 7 options, statistical inference would have been impossible. But thanks to his hack incompetence, this IDolator has an opportunity to learn something: his view of how science bloggers view ID and creationism is incorrect. All respondents chose "none of the above." That's real data.

A scientist confronted with unexpected data goes back to the drawing board. From the responses, he now has a better sense of how we actually see ID. He can write a better survey, if he actually wants reliable data.

But I suspected before and I suspect now that the point is less pure than the hack lets on. I suspect that almost any set of answers will be used to attack and mock, rather than to inform.

There was the hackitude following a visit to the KCFS forums, in which there was no serious effort at conversation, just stirring things up to use the provocation as an excuse to write about how the scientists are so bad. The same thing happened when a story from the hack website was rejected by a Skeptics' Circle host. Somehow, the decision by a private website operator not to post creationism became evidence that peer review was flawed.

What's the hypothesis, what are the predictions? How would this survey advance knowledge? If the answers to these questions aren't clear, it isn't an application of the scientific method.