Wednesday, September 28, 2005

Dembski on probability

Billy D., the Newton of something or other, posted a couple of glaringly incorrect pdfs.

One is an attack on Bayesian inference, which, in the course of misunderstanding the straightforward statistical process of hypothesis testing, also introduces blatant errors regarding the relationship of frequentist statistical techniques and Bayesian methods.

Without trudging through the details, I'll try to give a quick guide to the differences. While the Wikipedia argues that the Bayesian position can be seen as a superset of frequentist statistics, the two schools are very different in their underlying principles, and strict adherents to either school can be dogmatic in their defense of their approach. A frequentist holds that one can reject a hypothesis when the probability of an empirically measured result given a particular hypothesis is sufficiently small.

A Bayesian doesn't buy the rejection argument. You have "degrees of belief" and you accept the more likely of some number of hypotheses, conditional on future data.

A frequentist will generally evaluate a null hypothesis of "no effect," and if it's rejected, conditionally accept the alternative. So, if I reject the hypothesis that the slope of mean annual temperatures over the last century has been less than or equal to zero, I conditionally accept the hypothesis that temperatures have risen. In particular, I evaluate the probability of getting a particular dataset (D) if a given hypothesis were true (in the jargon, P(D|H). That's the probability of D given H). If I reject the hypothesis that temperatures would rise by a the measured amount if certain physical processes, and no others, were operating, I can conclude that some other process was operating, but not what that process is.

Bayesian stats work differently. I begin with some prior (non-zero) probability that the slope is greater than 0 (P(H)), then I gather new information (D). The probability that temperatures have risen is dependent on the prior probability (P(H)), the probability that you'd get the data you have if the temperature were rising (P(D|H)) and the probability you'd get your data if it weren't (P(D|~H)) (you must enumerate each of the alternate possibilities). You then choose the outcome which has the highest degree of belief. Not being much of a Bayesian, I may have mangled a detail here.

I've sketched out the way that a Bayesian might approach design detection, and it frankly makes sense to me. You incorporate the probability that a designer exists a priori, then you evaluate the probability that the designer would produce some data compared to the probability that other causes (natural selection, random weathering, whatever) would produce the same data, along with the probability of those processes occurring. Since the probability that evolution occurs is 1 (even Dembski doesn't disagree), but the probability of a designer existing (absent circular invocations of design) is a purely theological question, it's hard to know what to do. Dembski believes, deep in his heart, that a designer exists. Others assign that a probability of zero, in which case design is irrelevant. If we could pick a prior probability of a designer, we'd still need a coherent model of its capabilities and desires to be able to decide how likely it is that the designer would have made things a particular way.

Dembski dislikes that approach, since he really doesn't want to get into theological arguments. To avoid the Bayesian approach, he offers a frequentist approach which is fundamentally flawed. He divides all phenomena into three categories: Law, Chance, Design. By evaluating the probability of a given dataset, one can reject each of the first two, leaving only Design. Since it's generally impossible to reject all possible hypotheses of a given sort, we already have trouble. The probability of an event is dependent on a model of the process even under frequentist statistics, meaning he can't just look at the data, but he must evaluate P(D|H). The likelihood changes depending on the hypothesis. To reject all chance hypotheses or all law based hypotheses requires enumerating every possible such hypothesis and rejecting it. There's just no other way. This also leaves Design in the unique position of never having to prove itself. Convenient for Dembski, useless for everyone else.

Dembski concludes:

The Bayesian approach to statistical rationality is parasitic on the Fisherian [frequentist] approach and can properly adjudicate only among hypotheses that the Fisherian approach has thus far failed to eliminate.
As you can see, this is simply false. Bayesians reject the frequentist obsession with rejecting hypotheses. I doubt that it would be possible for a Bayesian to have a high degree-of-belief in a hypothesis which a comparable frequentist approach would reject, but that's because they are both bound by the same universe. In general, you'd expect both to give comparable results.

He babbles about specification and how it figures in, but who honestly knows what specification is supposed to mean. Sometimes it's a very specific thing, sometimes it seems like a random hand-wave.

Technorati Tags: ,