I’ve been arguing with myself again. I really liked that phony chocolate study because it so effectively demonstrated a couple of problems I tell my students about, so it’s a spectacular way to illustrate p-hacking and the unreliability of peer review. But as I was thinking about it, and how to present it to a class, it started to sink in that it also raises brand new problems that make it very difficult to use as an example. And then I started reading some other articles that emphasize the ethical concerns in this study.
Here’s Rachel Ehrenberg:
Bohannon and his colleagues decided to create a wrong to prove that wrongs exist. They lied to the public to make their point. Granted, it’s unlikely that anyone will be harmed by eating more dark chocolate. But not only does the caper do a disservice to people who are desperate for meaningful information about health and nutrition, it also undermines all of science and all of journalism. There’s real wrongdoing in both science and journalism (most infamously, see Stephen Glass, Jayson Blair, Janet Cooke, Jonah Lehrer, Brian Williams). But intentionally creating wrong to make a point is both bizarre and potentially very damaging.
“Our key resource as journalists is credibility,” Edmonds told me. “And a deceptive ploy like this could damage that.”
“Don’t lie” is a rather fundamental principle in science.
Another ethical principle is to do no harm, as Chris Lee of Ars Technica points out.
The end of the experiment is that millions of people all over the world were told that chocolate will help them lose weight. The consequence is that all those people who search (in vain) for fad diets—often to help them with their self-image—have been given yet another false data point and another failure to reflect upon.
In terms of ethical analysis, this is an experiment that did not tell us anything that wasn’t known already. On that score alone, the experiment fails to pass muster. Then there are the downsides. The reputation of science journals and science communicators just got a slight additional tarnish. Worse yet, there are people out there who have been taken in by the false reporting, and many of them will never know that the story was false from the beginning.
So if I were to present Bohannon’s fake chocolate study to a class, I’d have to say something like, “Here’s a really vivid example of using misleading statistics and getting the work published and well known. Don’t you ever, ever, ever do this.” That’s a problem.
But like I said, I argued with myself. It’s such a strong example by design — can I salvage it for instructional purposes?
I don’t think I can. What settled it for me is the personal perspective. I have a lab! I work with undergraduates, and I encourage them to try out new ideas. That means I have lots and lots of what I would charitably call ‘preliminary data’ — less charitably, you might call it half-assed data — that needs a lot more work to be useful. Bigger sample sizes, correcting for known confounding variables, just general replication…getting a student to focus for a year on something in that mess and make it a bit more robust is necessary before I could say it was really publishable.
But hey, what if I were to fish around in the data and look for something potentially cool and just fiddle with the stats and publish it in some low quality journal? Especially…what if I were to tell myself I was just doing it to highlight a problem in in science reporting? Wouldn’t that be a service?
And I find myself actively repulsed by the very idea. I am far too conscious of the shortcomings of that data — it’s tantalizing enough that it would be productive to put a student to work on the details, but it could be a dead end, or it could be accounted for by a trivial variable, or it could just be completely wrong. I’m a little surprised at how queasy even the thought of trying something like that made me feel, no matter what the excuse. I could not have assisted in carrying out Bohannon’s study, for instance — I would have begged off on personal ethical concerns.
When I put it in those terms, whether this was a study I could have participated in in good conscience, it suddenly makes it extremely problematic to present it to students, no matter how illuminating the results are. Science is all about the process, not the answer, and corrupting the process further, even to demonstrate the existence of extant corruption, is a violation. I tell students over and over that getting a pre-determined answer in lab isn’t the point — it’s how you get that answer that matters most. It’s a tough lesson to teach. I’m always getting distressed students who think they’re going to fail a course because their experiment didn’t work, and I have to tell them I don’t care that it didn’t work — do you understand why it failed? Do you have ideas for how to correct the problems? Did you learn something? Are you willing to accurately report what went wrong? That’s what I care about.
So how can I tell them that in this case, the end justifies the means?