Last year, the Elsevier journal Food and Chemical Toxicology by Gilles Seralini and others that purported to show that rats fed genetically modified corn were more prone to get cancer. The cranks loved it; Mike Adams thought it was great, it was touted on the Dr Oz show (I don’t know why they were concerned; these are the people who think cancer can be cured with herbs, urine, and drinking hydrogen peroxide).
But right from the beginning, scientists were appalled — not by the conclusion, but by the incredibly shoddy protocol used by the researchers. Biofortified went through the paper, step by step; would you believe that in a study with a control group and multiple experimental groups fed on GMO corn, the authors did not use any statistical tests to tell if there was a significant difference between any of the groups?
Let that sink in.
Here’s the first figure from the paper, and Ashley Ng’s breakdown of the data.
In the study, Figure 1 shows Kaplan Meier plots the number of rat deaths by “control group” and other “treatment groups”.
What do these mean? Well, not much because the authors failed to use a statistical test to tell if there was a difference between the control groups and treatment groups.
This is important, as all their claims relate to the incidence of cancers (and other “diseases”) in the “treatment group” compared to the “control group”. These comparisons can only be made if a statistical test shows that what you observe is not happening by chance.
Still on Figure 1, we see that several “treatment groups” of male rats receiving GM NK603 corn (the 22% group and 33% group) actually had fewer cancers than the male control group.
Similarly, a treatment group of male rats receiving 33% GM corn and Roundup had no difference to the control group, and two treatment groups receiving Roundup (A and C) had the same or less incidence of cancer compared with the control group.
I just eyeball the data, and what I see is typical noisy cancer mortality data (these are rats with a genetic predisposition to get cancer: 70% of males and 87% of females get it.) The one thing that would have looked significant to me is the higher likelihood of females coming down with cancer…but that’s a predilection already built into the strain. The problem is compounded by very small ns — there were only 20 rats in each group. I wouldn’t be surprised if the researchers had done some statistical analysis, but didn’t report it because the paltry statistical power of their study meant nothing was significant.
At the time the paper came out, Carl Zimmer also raised holy hell because it was another case of science by press conference. There were all kinds of complaints by scientists about the study, but journalists who got the paper in advance had to sign confidentiality agreements that prohibited them from consulting with experts — they were expected to flounder about in the dark and simply accept what they were told.
Here’s a little good news, though: the paper is being retracted. The editor-in-chief of the journal has made a rather weasely statement denying any wrong-doing by the authors, but that the paper is being retracted solely because of the ambiguity of the results.
Unequivocally, the Editor-in-Chief found no evidence of fraud or intentional misrepresentation of the data. However, there is legitimate cause for concern regarding both the number of animals in each study group and the particular strain selected. The low number of animals had been identified as a cause for concern during the initial review process, but the peer-review decision ultimately weighed that the work still had merit despite this limitation. A more in-depth look at the raw data revealed that no definitive conclusions can be reached with this small sample size regarding the role of either NK603 or glyphosate in regards to overall mortality or tumor incidence. Given the known high incidence of tumors in the Sprague-Dawley rat, normal variability cannot be excluded as the cause of the higher mortality and incidence observed in the treated groups.
He then goes on to praise the peer-review system, which is weird, because here’s a paper with huge obvious holes that fell right through the system. And furthermore, it’s a paper with gigantic political implications — right now, it is the linchpin of anti-GMO movements around the world — and should have gotten extra-careful scrutiny.
Jon Entine at Forbes has an especially thorough dissection of the implications of the paper. It reveals that other problems have emerged.
“The study appeared to sweep aside all known benchmarks of scientific good practice and, more importantly, to ignore the minimal standards of scientific and ethical conduct in particular concerning the humane treatment of experimental animals,” concluded a prominent group of scientists in Transgenic Review. They noted the rats in the study were exposed to extreme and unnecessary cruelty. None of the results depended on the size of their tumors or how long they lived after the tumor appeared. This unethical treatment of animals was a direct violation of accepted research protocol and was by itself grounds for the article being rejected initially or withdrawn.
It was rather peculiar that the paper reported only on mortality. They were studying the appearance of cancer, so a more relevant and direct measure would have been to assess by the appearance of tumors of a particular size, and then to humanely euthanize severely affected animals. This study had them languish in a cage until they died and could be scored. There was no description of the cancers in the control group! They did seem to have a number of rats with huge, grossly disfiguring tumors that were handy for photo ops, though.
So it was a terrible, sloppy paper with gaping deficiencies that somehow slipped past peer review but made scientists gape in surprise when they finally saw it published, and it’s finally being retracted. But too late: anti-GMO propagandists are now seeing the retraction as a sign that there is a conspiracy to Hide the Truth™, and are using the efforts to apply standards of evidence to the work as proof that Big Science is out to give everyone cancer.
Wait? Big Science isn’t out to give everyone cancer? Are you telling me attempts to modify crops are just to attempts improve yield and/or quality? I hate to tell you, it ain’t so. I looked at Big Ag’s and Big sciences business plan.
1. Modify the genome, and give cancer to everyone.
3. ????????
4. Profit!!!!
But seriously how could a paper with no statistical analyses make it to press. Some heads ought to roll.
The Black Stegotetrabelodon of Famine.
I knew about the sloppy protocol and the lack of statistical analysis, but I missed the part about the animal cruelty. I’m not surprised. Quacks routinely cut corners with the ethics of human experimentation, after all.
I’m with Skeptico on making a rule. Anyone who cites this study has forfeited the debate, either out of gullibility or dishonesty.
This doesn’t say much for the quality of this publication or the competence of its editors that they would accept a paper that make claims based on no statistical analysis.
I know several people who will be receiving some of the links within this article immediately. Thank you.
And shame on the journal that published the study in the first place.
It’s really appalling to see this kind of thing. I’m surrounded by some very talented, diligent and hard working scientists who can’t crack past the post-doc ‘phase’ of their career. They struggle to get really interesting work finished because of technical problems but they are honest about it and don’t cut corners. They compete for ever decreasing access to research opportunities not only amongst themselves but with underperforming doofuses who’s only talent is constructing a good story – you know what I mean. Then they get slapped in the face by seeing this sort of thing.
It’s not like the journal Food and Chemical Toxicology is some crappy journal that gets ignored. It’s impact factor is a modest 3.215.
That means it gets read.
And it means it is believed. I wont even begin to dismantle the paper based on my own assessment of its flaws. I think I’ll just get back to my own lab work, skip the statistical analysis and find the most biased strain of life form that will automatically show something interesting regardless of control v. experimental treatment. Hell, I could be a professor before the new year!
Where do these people find editors and reviewers this willing to publish crap?
I participate in the peer review process as a reviewer. Nowadays, I look *first* at the stats in any paper crossing my desk. I don’t have the energy any more to complain about scientists who can’t write coherently, when so many scientists fail to understand even the most basic statistics. Sometimes people actually DO know which analysis they should be doing, run them, don’t find significance and so run an inappropriate analysis (I’ve actually had a couple outright admit this during the review process! And no, those two didn’t get published… at least, not with *my* approval…).
How do we get a ‘statistical review’ step added to the peer review process? And no, that box some journals have for “statistical analysis appropriate yes/no” doesn’t count if it’s checked by someone who doesn’t understand stats themselves! Is there a shortage of statisticians or something? Shouldn’t journals at least have as many statisticians on staff as they do copy editors?
@5 ibbica
Yes, exactly. There is a remarkably poor understanding of stats out there.
How did this paper get published in the first place? The editor didn’t look for the data that showed a difference before sending it out for review and then at least two other reviewers didn’t understand what they should be looking for either. One of the things that should run through any reviewers mind is the following…. “If I don’t understand statistics, maybe I shouldn’t review an article with numerical data in it”.
Does that limit some reviewers ability to review papers? Awwww you poor thing…. LEARN THE FUCKEN TOOLS OF THE TRADE and learn statistics.
Its being a complete pile of garbage haven’t stopped Jenny McCarthy and her gang from using the Wakefield paper to convince credulous people into not vaccinating their kids, so unfortunately I don’t see the retraction having any effect on the frankenfood crowd.
@#5 ibbica
Journals have copy editors??!!
Cool, their method saves so much work! In my next publication, I’ll just plot some LHC data against a simulation and claim the discovery of the Alex boson based on the fact that the distributuon “really does look a bit like there’s a bump, doesn’t it”. I’m going to be famous!!!
I’ve started bookmarking your blog posts on these kinds of retractions so that when I inevitably see those studies touted I don’t have to fumble around in google to come up with a good response.
in other words the study as presented did not show much of anything.
sounded like it was done on the cheep then.
Would it be useful to re-do the study with a very much larger number of test animals and use more comprehensive analysis of the resulting data? Maybe even use different strains of rats for comparison?
It sounds like a subject that it might be wise to understand in depth, especially with regards to pesticide resistance and the likelihood of increased crop contamination as result.
uncle frogy
Scientific journals do not publish fiction, ergo that’s not actually a journal. Retraction doesn’t save matters at all; the error is far too egregious for that. Everyone who had anything to do with that atrocity needs to be fired. No one who actually read it did not know its “conclusions” were fiction.
now, see, this is what “bad science” looks like.
Yes, yes, a million times over, yes, though your statement comes uncomfortably close to the No True Scotsman fallacy.
Let Elsevier be forgotten
just a minor fisk of the fisking: a statistical test can never “show” that a result is or is not “happening by chance”, but can only assign probabilities.
PZ:
It’s also peculiar that they intentionally confounded their results by ending the study when the rats were old and dying anyway. Cancer happens!
What if somebody showed one of the following which since they are different are likely to have different effects in human beings?
1) A non-GMO artificially selected variety of X caused cancer (or some other health problem) at a higher rate than a different artificially selected non-GMO variety of X.
2) A non-GMO artificially selected variety of X caused cancer (or some other health problem) at a higher rate than a non-GMO variety of Y.
3) A wild version of A caused cancer (or some other health problem) at a higher rate than a wild version of B.
4) A wild version of A caused cancer (or some other health problem) at a higher rate than a GMO version of A.
Or any number of other combinations. I understand the need for good science and rigorous studies but I’m confused about what the impact of demonstrating some actual difference would be anyway. I’m quite certain that eating or not eating any number of foods will affect cancer rates and numerous other health issues. Who disputes that? So what? That’s just part of the cost / benefit of deciding what you should eat. I fail to understand why GMO or not is an issue. I’m pretty sure activist would still say in the case of 4) that you should eat the “natural” wild variety instead of the GMO version.
How many of those science loving Europeans are washing their organic fruit with a cigarette hanging from their lips?
Short answer – because its a bit like saying you don’t like anything a church does, but the communion wafers, and the silly nonsense that goes with them, isn’t that big a deal. I.e., all the other stupid BS going on, which is embraced by the same people, is a problem, but GMO paranoia isn’t.
Long version – Because.. The same dumb Fs that think GMO is “dangerous” because “unnatural”, think that herbs (like natural dude!), will cure you, and/or actually pump themselves full of shit which extra vitamins in them, because they don’t comprehend that those **do** cause cancer, in many cases (I recently saw a product, and shuddered, which, along with magical “berry extract” included 417% of the daily recommended vitamin C – the very thing which the quack founder of the mega-vitamin movement, along with his wife, was taking, even as evidence started showing up that it was dangerous in large amounts, before they *both* died from vitamin C caused cancer.)
The only way to fight this sort of stupidity, and ignorance, is with facts, and without them, we also don’t undo the BS legislation, which even today, protects the $46+ billion dollar gibberish industry (mind, this was the number 3 years ago, or so, and who knows what the numbers are now), so it can “fight against the evil $300 billion dollar ‘big pharma!'”. I keep wondering how “big” the gibberish industry has to get for it to be “big enough” that people start being just as scared of it, instead of imaging it to be some grand savior. Its a bit like praising the wacko that shows up with one really huge bomb, which can kill millions at once, because at least he only has “one” of the things, while the other side, presumably, had thousands, and is, thus, theoretically, more dangerous. The point that they are both a problem… And, every day I see totally mad new things on the shelf, claiming that X product is now Gluten free, or non-GMO, or “organic”, or “super food”, while completely ignoring the sheer idiocy of everyone on of this statements when, for example:
1. The “gluten free” this is some bloody product that hasn’t come without 50 miles of anything with gluten in it in the first place.
2. Is something, like a banana, that is cloned, or some other thing that has been engineered, the, “random and hope the next generation doesn’t have a mutant gene for something dangerous in it”, way.
3. Something like coconuts – As far as I am aware there isn’t a single damn thing they “spray” those for, nor it is, generally, *necessary* to bloody fertilize them, given how picky they are about climate and soil to start with, making it impossible to even grow some things unless they already grow there. (I could be wrong, but.. if water grew on trees, some idiot would be claiming that H2O could be “organic”.)
or
4) Its got every single one of the same bloody things that stuff on the shelf does, with maybe some “tiny” more amount in some random vitamin, or something, but is a) exotic, b) rare, c) not something most people bothered to buy before, and d) can be sold at a 500% mark up, or more, if it has to be imported from some lucky group of South American goat herders, who some kook concluded lived longer eating a special whatsit off the local plants, instead of because they ate half as much as we do, worked 4 times as hard, and hadn’t *yet* been introduced to tobacco, high fructose corn syrup, or vending machines.
It matters because in the last 20 years we have gone from a world, at least in the US, void of snake oil, due to a series of laws banning it, to one where its actually illegal, unless it kills someone, again, at least in the US, to even go, “Umm, heh, you know.. maybe you should actually prove that new thing doesn’t have stuff in it that is dangerous, and we just don’t know about yet, and does what you say it does, and stuff, before you sell it!”, if you are the FDA, or anyone else who is “supposed to” have authority to do that sort of thing. It seems, in a lot of Europe, they either never got around to actually stopping this stuff, to begin with, or, possibly, just followed along and threw out what ever laws they had, at more or less the same time. I admit, I am not sure which, but given the rabid hatred the US government seems to have to the idea of doing anything, “like those other people over there”, its hard to image, for me, that Europe hosed themselves, then we followed along (though, given the economics of it… profit might have overcome the natural hate of being like Europeans).
That needs to change, and, unfortunately, the only way it does, and real science gets back in the driver seat, is by ripping papers like this to shreds, when they get things wrong, and presenting better ones, that get it right, in order to actually work out what is really going on, instead of just letting every bloody idiot with a desperate wish that the new berry they discovered will “cure everything from boils to hair loss, to cancer, if you chant at it the right way, while shaking it in a bottle of water, and then stuffing it up your bung hole, or what ever “method” they come up with happens to be (or, is that one only used with coffee?)
Actually, holytape, if we cure every other disease but cancer, then everyone will eventually die of cancer. So, yes, I wouldn’t mind if science gives everyone cancer (by curing everything else).
citation sorely needed
Y’know Chas
since you are capable of copy-pasting and blockquoting
you could include the nym and number.
I could.
But usually I don’t.
I’m sorry if this displeases you.
I really doubt that you are sorry.
It would be less disrespectful if you actually made the effort.
Well, Chas, if you can’t be arsed to follow basic etiquette in your postings, why should anyone else be arsed to read them?
I ask seriously – if your answer is “I don’t care if they are” then it doesn’t make sense that you’d keep posting. So please don’t drag out that tired middle-school excuse for an answer.
Faaaaaaaaaaaiiiiiil.
For comparison, I can’t think of any paleontology journal right now that gets above 2. (Not counting more general journals in which paleontologists sometimes publish.)
A few still have one. Most don’t.
Dirty little secret: reviewers aren’t hired in the first place. We work for free, and usually we’re not even listed by the journal (that constitutes “being on the editorial board”).
Editors are “hired”, except almost none of them are ever paid either. (And those that are get symbolic sums, like 250 $/year.)
Plenty.
He does in fact have one of the things.
It’s imaginable that a stabilizer of radicals causes cancer in large doses… but I’m not aware of any research in that direction either.
Excuse me, that’s completely ridiculous.
If you really want to know who wrote the quote, which is after all completely irrelevant for the argument, copy it, press Ctrl+F, and paste it. I don’t understand what’s supposed to be disrespectful about not stuffing one’s comments with irrelevancies.
I do not understand that. I literally don’t.
The reason I’m so angry is that this
is a massive logical fallacy: . Have you no shame?
I actually did an experiment much like this one, with the result that not a single rat in the control group died of cancer. They all died of starvation a week after the experiment started.
Actually, it’s more like:
1. Persuade as many farmers as possible to use seeds they can only buy from you every year.
2. Make growth of the resulting crop dependent on a pesticide you also sell.
3. Prevent consumers having any information, hence choice, about the origin of their food.
4. Profit!!!!
Certainly BT Corn and Roundup Ready crops are a bad idea. There is evidence that these types of genetic modifications have encouraged resistance to BT and Roundup to evolve in the wild. Having more and more weed killer dumped in the environment is bad for the soil.
@21
He, and his wife, where taking something like 5,000 times the recommended amounts. Its not known, since the whole point of the passage of the legislation in the 90s was to “prevent” manditory testing of safety of such high doses, what is a safe amount.
http://www.dailymail.co.uk/health/article-2466592/Do-high-doses-vitamin-C-raise-prostate-cancer-risk.html
Also, for a better source, obviously:
http://www.quackwatch.com/01QuackeryRelatedTopics/pauling.html
The possibility that he died from it could be classed as speculation, if not for the fact that his wife died from cancer within months of him, and the single common factor between then, which might have been responsible, was is insane belief that 100% of all cancers could be cured with massive, insane, levels of vitamin C.
The biggest problem is that we have had decades of people believing that these things are both safe, and effective, no testing to prove it, and all of the “evidence” is from a few shoddy studies, which have, in the last maybe 10 years, started falling apart. In “smaller” amounts, but no one knows how much, extra vitamin C may be helpful, or not, depending on which cancer you are talking about, maybe.. but.. in actual fact, even then, its can be, as the American Cancer Society says, detectable, but not particularly significant, when taken as supplements (which is how most people take it):
http://www.cancer.org/treatment/treatmentsandsideeffects/complementaryandalternativemedicine/herbsvitaminsandminerals/vitamin-c
Eating foods containing it naturally, can cut the risk in half, but then.. that kind of implies that its either the combination of other things in them, or those other things, not the vitamin C itself that is producing the benefit.
What is known for certain is that Pauling and his wife didn’t appear to have any reason to have both contracted cancers, both declined at the rate they did, or died from them, within months of each other, other than his obsession with taking ludicrous amounts of something that no one had ever tested the safety of, in anything close to those levels. And, its common sense, or should be – toxicity is determined by how bloody much you take of something, not what it is. Worse, current research into anti-oxidants seem to imply that a) in normal amounts, they may help, some, in much larger amounts, they could possibly seriously screw up cellular repair, and that, ironically, at least some animals, which are being used to test such things, live longer without “any” in their systems, including the ones produced by their own bodies (the theory being that their presence triggers genetic repair systems, which are normally suppressed, especially if you are taking “extra” anti-oxidants). Its unknown whether you are better off with those repair systems working more, or if the errors that can accumulate, both due to oxidation, and mistakes in the repairs, are a worse threat. But, things are looking bad for the idea that you can live longer taking this stuff, never mind taking **lots** of it.
But, the key point here is, papers where put out, and claims made, just like in the article discussed here, which where, in some cases, preliminary, in others, predictive, but not definitive, and in other cases, actually total quack nonsense, and **despite** recent recommendations that people do not take vitamin supplements, if they don’t have deficiencies, since they are not known to actually help, and might increase some risks, instead, there are a hell of a lot of people, including ones that should know better, who keep telling people to take the bloody things.
This one brought me out of lurking.
@29:
Hi! If you’re being sarcastic then I apologize, but just in case you’re not:
1) Many farmers I know but their seeds, even non-GM every year. It’s easier and you don’t have to worry so much about consistency.
2) Herbicide resistant crops aren’t dependent on the herbicide to grow.
3) Labeling laws, as they’ve currently been pushed, wouldn’t provide useful information to consumers, nor have they been consistent about what must be labelled.
4) Farming is a business. If farmers were being squeezed by seed companies then they wouldn’t keep buying from them.
@30
Much like antibiotics, resistance can be avoided through proper use. Bt crops, for instance, need to be planted with a small section of non-Bt crops (the haven concept) to help prevent resistance. Roundup Ready can have the same misuse, and you’re right about weed killer, but glyphosate is one of the least worst herbicides out there.
I meant to say “buy” not “but”, “labeled” not “labelled”. Sorry!
@30: Anything we do to fight weeds and pests short of rapid extinction is going to make them to evolve countermeasures.
Meh, late to the party…
1. Sure, but do so by
a) Inserting a trait that is highly desirable to farmers in the first place
b) Run breeding programs which generate the most productive hybrids which are also highly desirable to farmers.
One must also keep in mind that the actual business model is vaguely this, but the GM traits are so broadly licensed that when discussing a succesful GM trait only a) really applies, and one is no longer discussing buying a company’s seed, but *any* seed that has the company’s trait present (this is why traits are so penetrant in the market – 96%+ of soy in the US may be GM, but any one company only has 30% or so of the actual market share for seeds.
2. Not so much really. RR crops are not dependant on the use of Roundup, although purchasing RR seed and then not using Roundup would appear to be an exercise in utter futility. Bt crops also are not dependant on any pesticides at all, so your fictional business model doesn’t make the least bit of sense here either.
3. Any information? I guess the GMO free project and USDA certified organic don’t actually exist and I’ve been hallucinating every time I’ve been at the supermarket of late. I really should stop dropping acid before purchasing groceries. It does make for an interesting pantry though. (also see somewhat less pithy response in 33 above…)
4. Indeed. Although I’m pretty sure there are too few exclamation points associated, nobody on wall street is going to get excited by so few.