I read with growing horror this account of the research practices of the Wansink lab. They do research in nutrition, or maybe some combination of economics, psychology, and dietary practices — it’s described as “research about how people perceive, consume, and think about food”, and it’s not stuff I’d ever be interested in reading (although that does not imply that it has no value). The PI, Brian Wansink, wrote up a summary of his process on a blog, though, and honestly, my jaw just dropped reading this.
A PhD student from a Turkish university called to interview to be a visiting scholar for 6 months. Her dissertation was on a topic that was only indirectly related to our Lab’s mission, but she really wanted to come and we had the room, so I said “Yes.”
When she arrived, I gave her a data set of a self-funded, failed study which had null results (it was a one month study in an all-you-can-eat Italian restaurant buffet where we had charged some people ½ as much as others). I said, “This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool (rich & unique) data set.” I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.
He described it as a
failed study with
null results. There’s nothing wrong with that; it happens. What I would think would be appropriate next would be to step back, redesign the experiment to correct flaws (if you thought it had some; if it didn’t, you simply have a negative result and that’s what you ought to report), and repeat the experiment (again, if you thought there was something to your hypothesis).
That’s not what he did.
He gave his student the same old data from the same set of observations and asked her to rework the analyses to get a statistically significant result of some sort. This is deplorable. It is unacceptable. It means this visiting student was not doing something I would call research — she was assigned the job of p-hacking.
Further, what’s just as shocking is that Wansink sees so little wrong with this behavior that he would publicly write about it.
He’s not done.
Every day she came back with puzzling new results, and every day we would scratch our heads, ask “Why,” and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them.
Note: no new experiments. This is all just churning over the same failed experiment, the same failed data set. Back in the day, I learned that you design an experiment to test a specific hypothesis, and that you don’t get to use the data to test different hypotheses until you get a result that you like. But what do I know, I’m old.
Still not done.
I outlined the first paper, and she wrote it up, and every day for a month I told her how to rewrite it and she did. This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).
Out of this one failed (I repeat, fucking failed) data set, they ground out FOUR papers. Four. Within a few months. Good god, I’ve been doing everything wrong.
You might be wondering what these papers were that he milked out of this failed data set. Here are the titles:
Lower Buffet Prices Lead to Less Taste Satisfaction
Peak-end pizza: prices delay evaluations of quality
Low prices and high regret: how pricing influences regret at all-you-can-eat buffets
Eating Heavily: Men Eat More in the Company of Women
I am trying hard not to be judgmental, and failing. These sound like superficial, pointless crap churned out to appease a capitalistic marketing machine, with virtually no value and making no contribution to human knowledge. But I guess it’s good enough to get you a leadership position in a prestigious lab at Cornell.
It’s also a huge problem that this kind of strategy works. It’s not just Wansink — it’s a science establishment that allows and even encourages this kind of garbage production.
I hear there’s a replication crisis in the sciences. I have no idea how that could be.