John Bohannon of Science magazine has developed a fake science paper generator. He wrote a little, simple program, pushes a button, and gets hundreds of phony papers, each unique with different authors and different molecules and different cancers, in a format that’s painfully familiar to anyone who has read any cancer journals recently.
The goal was to create a credible but mundane scientific paper, one with such grave errors that a competent peer reviewer should easily identify it as flawed and unpublishable. Submitting identical papers to hundreds of journals would be asking for trouble. But the papers had to be similar enough that the outcomes between journals could be comparable. So I created a scientific version of Mad Libs.
The paper took this form: Molecule X from lichen species Y inhibits the growth of cancer cell Z. To substitute for those variables, I created a database of molecules, lichens, and cancer cell lines and wrote a computer program to generate hundreds of unique papers. Other than those differences, the scientific content of each paper is identical.
The fictitious authors are affiliated with fictitious African institutions. I generated the authors, such as Ocorrafoo M. L. Cobange, by randomly permuting African first and last names harvested from online databases, and then randomly adding middle initials. For the affiliations, such as the Wassee Institute of Medicine, I randomly combined Swahili words and African names with generic institutional words and African capital cities. My hope was that using developing world authors and institutions would arouse less suspicion if a curious editor were to find nothing about them on the Internet.
The data is totally fake, and the fakery is easy to spot — all you have to do is read the paper and think a teeny-tiny bit. The only way they’d get through a review process is if there was negligible review and the papers were basically rubber-stamped.
The papers describe a simple test of whether cancer cells grow more slowly in a test tube when treated with increasing concentrations of a molecule. In a second experiment, the cells were also treated with increasing doses of radiation to simulate cancer radiotherapy. The data are the same across papers, and so are the conclusions: The molecule is a powerful inhibitor of cancer cell growth, and it increases the sensitivity of cancer cells to radiotherapy.
There are numerous red flags in the papers, with the most obvious in the first data plot. The graph’s caption claims that it shows a "dose-dependent" effect on cell growth—the paper’s linchpin result—but the data clearly show the opposite. The molecule is tested across a staggering five orders of magnitude of concentrations, all the way down to picomolar levels. And yet, the effect on the cells is modest and identical at every concentration.
One glance at the paper’s Materials & Methods section reveals the obvious explanation for this outlandish result. The molecule was dissolved in a buffer containing an unusually large amount of ethanol. The control group of cells should have been treated with the same buffer, but they were not. Thus, the molecule’s observed “effect” on cell growth is nothing more than the well-known cytotoxic effect of alcohol.
The second experiment is more outrageous. The control cells were not exposed to any radiation at all. So the observed “interactive effect” is nothing more than the standard inhibition of cell growth by radiation. Indeed, it would be impossible to conclude anything from this experiment.
This procedure should all sound familiar: remember Alan Sokal? He carefully hand-crafted a fake paper full of po-mo gobbledy-gook and buzzwords, and got it published in Social Text — a fact that has been used to ridicule post-modernist theory ever since. This is exactly the same thing, enhanced by a little computer work and mass produced. And then Bohannon sent out these subtly different papers to not one, but 304 journals.
And not literary theory journals, either. 304 science journals.
It was accepted by 157 journals, and rejected by 98.
So when do we start sneering at science, as skeptics do at literary theory?
Most of the publishers were Indian — that country is developing a bit of an unfortunate reputation for hosting fly-by-night journals. Some were flaky personal obsessive “journals” that were little more than a few guys with a computer and a website (think Journal of Cosmology, as an example). But some were journals run by well-known science publishers.
Journals published by Elsevier, Wolters Kluwer, and Sage all accepted my bogus paper. Wolters Kluwer Health, the division responsible for the Medknow journals, "is committed to rigorous adherence to the peer-review processes and policies that comply with the latest recommendations of the International Committee of Medical Journal Editors and the World Association of Medical Editors," a Wolters Kluwer representative states in an e-mail. "We have taken immediate action and closed down the Journal of Natural Pharmaceuticals."
Unfortunately, this sting had a major flaw. It was cited as a test of open-access publishing, and it’s true, there are a great many exploitive open-access journals. These are journals where the author pays a fee — sometimes a rather large fee of thousands of dollars — to publish papers that readers can view for free. You can see where the potential problems arise: the journal editors profit by accepting any papers, the more the better, so there’s pressure to reduce quality control. It’s also a situation in which con artists can easily set up a fake journal with an authoritative title, rake in submissions, and then, perfectly legally, publish them. It’s a nice scam. You can also see where Elsevier would love it.
But it’s unfair to blame open access journals for this problem. They even note that one open-access journal was exemplary in its treatment of the paper.
Some open-access journals that have been criticized for poor quality control provided the most rigorous peer review of all. For example, the flagship journal of the Public Library of Science, PLOS ONE, was the only journal that called attention to the paper’s potential ethical problems, such as its lack of documentation about the treatment of animals used to generate cells for the experiment. The journal meticulously checked with the fictional authors that this and other prerequisites of a proper scientific study were met before sending it out for review. PLOS ONE rejected the paper 2 weeks later on the basis of its scientific quality.
The other problem: NO CONTROLS. The fake papers were sent off to 304 open-access journals (or, more properly, pay-to-publish journals), but not to any traditional journals. What a curious omission — that’s such an obvious aspect of the experiment. The results would be a comparison of the proportion of traditional journals that accepted it vs. the proportion of open-access journals that accepted it… but as it stands, I have no idea if the proportion of bad acceptances within the pay-to-publish community is unusual or not. How can you publish something without a control group in a reputable science journal? Who reviewed this thing? Was it reviewed at all?
Oh. It’s a news article, so it gets a pass on that. It’s also published in a prestigious science journal, the same journal that printed this:
This week, 30 research papers, including six in Nature and additional papers published online by Science, sound the death knell for the idea that our DNA is mostly littered with useless bases. A decade-long project, the Encyclopedia of DNA Elements (ENCODE), has found that 80% of the human genome serves some purpose, biochemically speaking. Beyond defining proteins, the DNA bases highlighted by ENCODE specify landing spots for proteins that influence gene activity, strands of RNA with myriad roles, or simply places where chemical modifications serve to silence stretches of our chromosomes.
And this:
Life is mostly composed of the elements carbon, hydrogen, nitrogen, oxygen, sulfur, and phosphorus. Although these six elements make up nucleic acids, proteins, and lipids and thus the bulk of living matter, it is theoretically possible that some other elements in the periodic table could serve the same functions. Here, we describe a bacterium, strain GFAJ-1 of the Halomonadaceae, isolated from Mono Lake, California, that is able to substitute arsenic for phosphorus to sustain its growth. Our data show evidence for arsenate in macromolecules that normally contain phosphate, most notably nucleic acids and proteins. Exchange of one of the major bio-elements may have profound evolutionary and geochemical importance.
I agree that there is a serious problem in science publishing. But the problem isn’t open-access: it’s an overproliferation of science journals, a too-frequent lack of rigor in review, and a science community that generates least-publishable-units by the machine-like application of routine protocols in boring experiments.
