You may have heard that the replicability of much biomedical research has been called into question, in particular by the Ioannidis paper in 2005 that demonstrated that a heck of a lot of junk got into print, largely as a consequence of statistical noise being treated as significant, for a host of reasons. It was a bit of a wake-up call (unfortunately, most people just rolled over an smacked the snooze button), but one person who is on full alert is Dan Graur. Graur is being impolite again, and has a recommendation to improve the problem.
Interestingly, the rate with which junk claims are published in the field of experimental physics is nowhere near the stratospheric rates that are found in biology and medicine. Why the difference? Dr. Ioannidis thinks that there are two reasons for the difference. First, it seems that in the biomedical research community there exists an aversion to publish negative results, especially negative results of failed replications.
Second, it seems that there are sociological differences between the physics community and the biomedical community. In physics, there seems to be a higher “community standard for shaming reputations.” If people step out of line and make unsubstantiated claims, they are shamed in public.
Wait — I have to call a foul on the play. The Ioannidis paper certainly does say the first difference, but the second…nope. The quoted phrase about shaming doesn’t appear anywhere in the source Graur links to — I’d like to see where it came from.
I’m inclined to agree with it, except that I don’t have any evidence of any public shaming going on in the physics community. I’d like to know more about how physicists police their own than is given here.
This is the abstract from Ioannidis — you can easily see that there is a focus on removing bias and better statistics, there isn’t anything about using shame as a tool.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
But don’t let that stop Graur, he’s on a roll!
In biomedicine, the search for truth is no longer a virtue, politeness is. According to an editorial in Nature Methods that singled out our work for criticism, one should avoid “harsh and offensive words” at all costs. “Civility in discourse is essential,” proclaim the editors of Nature Methods. Do not shame reputations! Well… by not shaming reputations, we have built a field of study where bombast thumps substance, and where wasters of public money are rewarded. By paying attention to “manners” we have prostituted science to a degree where “most published research findings are false.”
Rudeness has nothing to do with science. Science is not about abiding by a code of behavior put forward by Miss Manners. In criticizing ENCODE, our style of writing was meant to bring to the attention of the public a problem generated by the ENCODE propaganda barrage.
Face it, ENCODE for creationists was like “water memory” for people believing in homeopathic medicine. ENCODE deserved the same treatment as the “water memory” paper that was published in Nature by Jacques Benveniste. ENCODE needed to be shredded to pieces in a manner similar to the way Great Randy [the Amazing Randi] shredded “water memory.” The Great Randy [the Amazing Randi] was so rude, that his criticism was likened by Benveniste to a “Salem witch hunt.”
In science, sometimes a strong rude voice is needed to fight self-promotion and self-delusion. My favorite example of a rude voice concerns Theodor Roosevelt and his refutation of Abbott Thayer’s theory on all coloration in nature being “concealing” (e.g., the famous flamingoes in the sunset). Thayer’s book was shredded to pieces by Theodor Roosevelt (one year after completing his presidency). I wish my mastery of the English language would allow me to emulate Roosevelt’s viciousness. Alas, English is my third language.
We need strong and impolite voices to fight “stem cells created by acid baths,” “cold fusion,” “arsenic-based life,” and other feats of self delusion. People still believe that Svante Pääbo sequenced ancient DNA from an Egyptian mummy in 1985. Why? Because there were no strong and impolite rebuttals. Every criticism was whispered and “soto voce.” Science has become a collection of Yes Men (and Women) afraid of the big shots and their own shadows.
I agree, and I’d like to see more vigorous responses to the boring lot of trivial phenomenology that is cluttering up some of the journals I like to read. We’re getting to a point where the literature is swamped with kipple that could benefit from some housecleaning, and a little less emphasis on publishing for the sake of publishing. But we rely so often on the quantity of articles published as a metric for academic success, rather than the quality.
But I’d also like to see a stronger analysis comparing the literature in physics and biomedicine — is it really that different?