On his show Last Week Tonight, John Oliver gives an excellent segment on the nature of science and how the relentless drive to hype it so as to provide sensationalist headlines has resulted in a highly distorted view of how it works. (Thanks to reader Jeff Hess for the tip.)
One of the things Oliver mentions is ‘p-hacking’ where, rather than designing an experiment to test a hypothesis linking two variables to see if any correlations between them are statistically significant, you mine already existing data to search for pairs of variables that are correlated statistically significantly and, if you find them, then publish just those results.
This whole issue of p-values is tricky. The Discover magazine blogger Neuroskeptic has produced a video that attempts to explain what it is.
Christie Aschwanden reports on a meeting of the American Statistical Association where 26 experts issued a statement that said that it is high time to stop misusing p-values because the consequences are serious.
The misuse of the p-value can drive bad science (there was no disagreement over that), and the consensus project was spurred by a growing worry that in some scientific fields, p-values have become a litmus test for deciding which studies are worthy of publication. As a result, research that produces p-values that surpass an arbitrary threshold are more likely to be published, while studies with greater or equal scientific importance may remain in the file drawer, unseen by the scientific community.
The results can be devastating, said Donald Berry, a biostatistician at the University of Texas MD Anderson Cancer Center. “Patients with serious diseases have been harmed,” he wrote in a commentary published today. “Researchers have chased wild geese, finding too often that statistically significant conclusions could not be reproduced.” Faulty statistical conclusions, he added, have real economic consequences.
One of the most important messages is that the p-value cannot tell you if your hypothesis is correct. Instead, it’s the probability of your data given your hypothesis. That sounds tantalizingly similar to “the probability of your hypothesis given your data,” but they’re not the same thing, said Stephen Senn, a biostatistician at the Luxembourg Institute of Health. To understand why, consider this example. “Is the pope Catholic? The answer is yes,” said Senn. “Is a Catholic the pope? The answer is probably not. If you change the order, the statement doesn’t survive.”
A common misconception among nonstatisticians is that p-values can tell you the probability that a result occurred by chance. This interpretation is dead wrong, but you see it again and again and again and again. The p-value only tells you something about the probability of seeing your results given a particular hypothetical explanation — it cannot tell you the probability that the results are true or whether they’re due to random chance. The ASA statement’s Principle No. 2: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”
The p-hacking process Oliver talks about is similar to the kinds of things that people like to believe in that seem to show that there are deep, mysterious underlying forces at work in life. For example, some may have seen this email that circulated some time ago showing the seemingly astounding similarities between the murders of presidents Lincoln and Kennedy.
Abraham Lincoln was elected to Congress in 1846.
John F. Kennedy was elected to Congress in 1946.
Abraham Lincoln was elected President in 1860.
John F. Kennedy was elected President in 1960.
The names Lincoln and Kennedy each contain seven letters.
Both were particularly concerned with civil rights.
Both wives lost their children while living in the White House.
Both Presidents were shot on a Friday.
Both were shot in the head.
Lincoln’s secretary, Kennedy, warned him not to go to the theatre.
Kennedy’s secretary, Lincoln, warned him not to go to Dallas.
Both were assassinated by Southerners.
Both were succeeded by Southerners.
Both successors were named Johnson.
Andrew Johnson, who succeeded Lincoln, was born in 1808.
Lyndon Johnson, who succeeded Kennedy, was born in 1908.
John Wilkes Booth was born in 1839.
Lee Harvey Oswald was born in 1939.
Both assassins were known by their three names.
Both names are comprised of fifteen letters
Booth ran from the theater and was caught in a warehouse.
Oswald ran from a warehouse and was caught in a theater.
Booth and Oswald were assassinated before their trials.
This kind of thing can seem highly impressive to those who are unaware how it is possible, if time consuming, to sift through the vast amounts of data that are associated with any real-life event and extract just those few that fit a theory. Snopes has done a good job of showing the vacuity of the Lincoln-Kennedy coincidences.
Trying to give the general public an idea of how science actually works and when we can take scientific conclusions seriously and when we should be skeptical is the theme of the book that I am currently working on tentatively titled The Paradox of Science.