Compare:

We need somewhere to bury sloppy research on fast food, after all. Brian Wansink gets interviewed on Retraction Watch (y’all remember Wansink, the fellow who ground his data exceedingly fine to extract four papers from a null result), and he does himself no favors.

Well, we weren’t testing a registered hypothesis, so there’d be no way for us to try to massage the data to meet it. From what I understand, that’s one definition of p-hacking. Originally, we were testing a hypothesis – we thought the more expensive the pizza, the more you’d eat. And that was a null result.

But we set up this two-month study so that we could look at a whole bunch of totally unanswered empirical questions that we thought would be interesting for people who like to eat in restaurants. For example, if you’re eating a meal, what part influences how much like the meal? The first part, the middle part, or the last part? We had no prior hypothesis to think anything would predominate. We didn’t know anybody who had looked at this in a restaurant, so it was a totally empirical question. We asked people to rate the first, middle, and last piece of pizza – for those who ate 3 or more pieces – and asked them to rate and the quality of the entire meal. We plotted out the data to find out which piece was most linked to the rating of the overall meal, and saw ‘Oh, it looks like this happens.’ It was total empiricism. This is why we state the purpose of these papers is ‘to explore the answer to x.’ It’s not like testing Prospect Theory or a cognitive dissonance hypothesis. There’s no theoretical precedent, like the Journal of Pizza Quality Research. Not yet.

That last bit sounds like a threat.

Here’s the thing: we all do what he describes. An experiment failed (yes, it’s happened to me a lot). OK, let’s look at the data we’ve got very carefully and see if there’s anything potentially interesting in it, any ideas that might be extractable. The results are a set of observations, after all, and we should use them to try and figure out what’s going on, and in a perfect world, there’d be public place to store negative results so they aren’t just buried in a file drawer somewhere. There’s nothing wrong with analyzing your data out the wazoo.

The problem is that he then published it all under the guise of papers testing different hypotheses. Most of us don’t do that at all. We see a hint of something interesting buried in the data for a null result, and we say, “Hmm, let’s do an experiment to test this hypothesis”, or “Maybe I should include this suggestive bit of information in a grant proposal to test this hypothesis.” Just churning out low-quality papers to plump up the CV is why I said this is a systemic problem in science — we reward volume rather than quality. It doesn’t make scientists particularly happy to be drowning in drivel, but Elsevier is probably drooling at the idea of a Journal of Pizza Quality Research — another crap specialized journal that earns them an unwarranted amount of money and provides another dumping ground for said drivel being spewed out.

Wansink seems to be dimly aware of this situation.

These sorts of studies are either first steps, or sometimes they’re real-world demonstrations of existing lab findings. They aren’t intended to be the first and last word about a social science issue. Social science isn’t definitive like chemistry. Like Jim Morrison said, “People are strange.” In a good way.

Yes. First steps. Maybe you shouldn’t publish first steps. Maybe you should hold off until you’re a little more certain you’re on solid ground.

No one expects social science to be just like chemistry, but this idea that you don’t need robust observations with solid methodology might be one reason there is a replicability crisis. Rather than repeating and engaging in some healthy self-criticism of your results, you’re haring off to publish the first thing that breaches an arbitrary p-value criterion.

There really are significant problems with the data he did publish, too. Take a look at this criticism of one of his papers. The numbers don’t add up. The stats don’t make sense. His tables don’t even seem to be appropriately labeled. You could not replicate the experiment from the report he published. This stuff is incredibly sloppy, and he doesn’t address their failings in the interview, except inadequately and in ways that don’t solve the problems with the work.

Again, I’m trying to be generous in interpreting the purpose of this research — often, interdisciplinary criticism can completely miss the point of the work (see also how physicists sometimes fail to comprehend biology, and inappropriately apply expectations from one field to another) — but I’m also seeing a lack of explanation of the context and relevance of the work. I mean, when he says, “For example, if you’re eating a meal, what part influences how much like the meal? The first part, the middle part, or the last part?”, I’m just wondering why. Why would it matter, what are all the variables here (not just the food, but in the consumer), and what do you learn from the fact that Subject X liked dessert, but not the appetizer?

It sounds like something a restaraunteur or a food chain might want to know, or that might might appeal to an audience at a daytime talk show, but otherwise, I’m not seeing the goal…or how their methods can possibly sort out the multitude of variables that have to be present in this research.

There’s a popular scam among creationists: they offer big prizes to anyone who can “prove” that evolution is true. They never award these prizes, and I suspect they usually don’t even have the cash on hand, because they’ve got an ace up their sleeves. They set unreachable criteria. For example, to win Joseph Mastropaolo’s evolution prize, one must present evidence that persuades a team of judges — judges who are hand-picked by Mastropaolo. I think the game is stacked.

Now look who is playing a similar game: Robert Kennedy Jr. and Robert De Niro are offering a $100,000 prize for proof that vaccines are safe. I don’t quite know what they expect would constitute proof, since they seem to disregard the extensive clinical trials that have been carried out, or the lack of significant numbers of dead babies from their shots (almost 90% of infants get a thorough series of vaccinations, yet somehow we don’t have piles of dead babies), or the historical evidence (visit a 19th century graveyard, and you will find those piles of dead babies…modern graveyards are mostly full of old dead people), or the remarkable improvement in public health with the introduction of, for instance, the polio vaccine, and the effective eradication of smallpox, or that measles kills about 100,000 people a year (but very few in the US), all of which would be preventable by vaccines.

You know, that $100,000 prize would help a lot in vaccinating all the people in Asia and Africa who are suffering from measles — about 20 million people each year.

So what are the criteria for winning this prize?

Kennedy explained that the WMP will pay $100,000 to the first journalist, or other individual, who can find a peer-reviewed scientific study demonstrating that thimerosal is safe in the amounts contained in vaccines currently being administered to American children and pregnant women. Kennedy believes that even “a meager effort at homework” will expose that contention as unsupported by science.

Hold your horses, everyone! <rushes to the PubMed link I always keep handy>

Early exposure to the combined measles-mumps-rubella vaccine and thimerosal-containing vaccines and risk of autism spectrum disorder.

CONCLUSIONS:
No convincing evidence was found in this study that MMR vaccination and increasing thimerosal dose were associated with an increased risk of ASD onset.

Administration of thimerosal-containing vaccines to infant rhesus macaques does not result in autism-like behavior or neuropathology.

These data indicate that administration of TCVs and/or the MMR vaccine to rhesus macaques does not result in neuropathological abnormalities, or aberrant behaviors, like those observed in ASD.

Vaccines and autism in primate model.

Administration of thimerosal-containing vaccines to infant rhesus macaques does not result in autism-like behavior or neuropathology.

GIMME MY MONEY, BOBBY². Those were a few papers that turned up in just the first page of a search — there were 180 more pages, but I didn’t bother looking, because I like the idea of winning a couple of years’ salary for the minimal amount of work. I wonder — could I be even lazier and just send them a link to PubMed?

I suspect that I won’t get paid, because there are other, mysterious excuses they’ll have for rejecting the evidence, just like the creationists do. The stated criteria are just to obvious and simple and have been met over and over in decades of peer-reviewed research.

They won’t pay up, because like the creationists, they’re only going to accept ‘evidence’ that supports their presuppositions, and the purpose of the reward is not to get information delivered to them — that information is freely available already — but to promote a lie. “We offered all this money, and no one could provide evidence, therefore you know that evolution/vaccines are false!” It’s a pretty tacky tactic.

It’s sad, too. I’ve liked many of De Niro’s movies. Now I’ll never be able to watch them again without being conscious that the actor is a colossal dumbass.