Thursday, October 27, 2005

Polling questions

In my previous post on polling, I said that FDM 's authors "propose a bunch of questions that no actual pollster would actually ask." What are these questions?
A question asking about a respondent’s party affiliation would be asked something like this…

Which political party are you registered to vote with?

A question on a respondent’s party ID would be something like this.

Regardless of your registered party affiliation, do you generally identify with which political party?

While I showed that these are not actually the questions pollsters ask about party, why am I so adamant that pollsters would never ask them?

The problem is, these are open-ended questions. While those are useful in some circumstances, those questions are prone to biases and the vagaries of the respondent. (That SUSA's computerized calling system doesn't handle such questions only shows how little FDM knows about what the issue at hand.)

Each of the example questions I posted were identical in offering people the option of "considering themselves" (the term all three use) Republicans, Democrats, or Independents, with Harris offering a "some other party" option, and the LA Times poll revealing "something else" and "don't know" as other options.

By limiting people's options, you avoid complicating factors like Minnesota's DFL, or people who might volunteer fringe parties if asked, but will pick a major party if prompted for those only. You ensure consistent results, minimize human error, and adhere to standard practices in the field by making questions multiple choice.

It seems like a small thing, but any pollster you ask (and I encourage FDM to try it before embarrassing themselves further) will tell you that the precise wording of a question can alter results, as can the precise order of the questions. That FDM is uninterested in such basic concerns shows just how out of their depth they are.

This matters because bitching about polls like FDM is doing is a minor but typical example of the Republican war on expertise. Not all Republicans are part of this war, but those who are, deserve nothing but scorn. Just as Senator Inhofe's belief that he's a better judge of the state of climate science than the community of climate scientists, FDM think they know better than the community of pollsters how to handle partisan identification.

Mystery Pollster (a professional himself), writes that:

neither CBS/New York Times, ABC/Washington Post,* Gallup/CNN/USA Today, Time, Newsweek, The Pew Research Center and nor the Annenberg National Election Survey weights their results by Party ID

As we showed before, party ID is the same as party affiliation, etc.

In addition, the National Council on Public Polling writes that weighting by party ID is "little better than a guessing game where the pollster is substituting his or her judgment for scientific method."

Indeed, the NCPP specifically rejects FDM's proposal of weighting by the party ID of likely voters. Weighting by party ID of the population at large is more appropriate, but still sits in their category of "Bad Weighting."

Since they claim to be preparing a post on how they want to weight by party ID (which is the same as party affiliation), these are things I hope they keep in mind.