Wednesday, June 15, 2005

Don't know much about statistics

The Heritage Foundation blows it, and the New York Times runs a bad headline. Heritage reanalyzed data which shows that people who take virginity pledges (a common component of abstinence-only classes) are 4 times more likely to have anal sex, 6 times more likely to have oral sex, have sex without contraception, have STDs, get pregnant, and have more abortions.

Heritage claims they got the opposite results, but only by cooking the numbers, and using flawed data.
The new findings were based on the same national survey used by earlier studies and conducted by the Department of Health and Human Services. But the authors of the new study used different methods of statistical analysis from those in an earlier one that was widely publicized, making direct comparisons difficult.

Independent experts called the new findings provocative, but criticized the Heritage team's analysis as flawed and lacking the statistical evidence to back its conclusions. The new findings have not been submitted to a journal for publication, an author said. The independent experts who reviewed the study said the findings were unlikely to be published in their present form.

The team needs to do "a lot of work" on its paper, said David Landry, a senior research associate at the Alan Guttmacher Institute in New York. He said in an interview that it was "a glaring error" to use the result of a statistical test at a 0.10 level of significance when journals generally use a lower and more rigorous level of 0.05.

Dr. Johnson, a co-author, defended the team's methods and said many journal articles used the higher level and let readers decide the merits of the findings.

Mr. Landry also criticized the Heritage team's reliance on self-reports of sexually transmitted diseases among those who took the pledge, saying that group would be less likely to report them. "The underreporting problem is so severe that it makes that data highly questionable," Mr. Landry said.

Dr. Bearman said: "Our analyses showed that pledgers are less likely to get tested for S.T.D.'s, be diagnosed as having an S.T.D. and to see a doctor because they are worried about having an S.T.D. Most S.T.D. infections are asymptomatic, and therefore, people don't know that they have an S.T.D. unless they get tested. The use of self-report data for S.T.D.'s is therefore extremely problematic."

Mr. Landry and Dr. Freya Sonenstein, who directs the center for adolescent health at the Johns Hopkins Bloomberg School of Public Health, urged the Heritage team to try to publish its findings.
Journals may accept papers with some results at 0.1, if they have other compelling results. Offering p-values and letting the reader decide is often a wise choice. However, if you do ten analyses where no statistical relationship exists, you have a good shot at getting one test showing an effect at 0.1. I bet they ran more than 10 tests, so you have to wonder about the validity of the tests.

And here's the thing. If you can show that virginity pledgers with an STD are less likely to find out that they are infected, isn't that a pretty good argument against the pledges? After all, there is either no difference in infection rates, or the rates are higher in pledgers.

Untreated chlamydia leads to infertility, higher HIV infection rates, and ectopic pregnancies. Ectopic pregnancy can be deadly.

Therefore, virginity pledges can be deadly.