I’ve been talking to my class this week (and it’s going to be a theme next week) about the difficulty of analyzing epidemiological data on cancer — that there are so many steps to cancer progression and so many environmental and genetic inputs to the disease that sorting them all out is extremely complex. What I haven’t mentioned yet, but definitely will now, is the factor big money plays in encouraging statistical fraud.
The University of Texas MD Anderson Cancer Center in Houston has been the top-ranked cancer center on US News & World Report’s best-of list for the past 7 years. But that top ranking was aided by a massive error in data used to evaluate its care.
The error in M.D. Anderson’s favor was made by–M.D. Anderson! Avery Comarow, who assembles the rankings at U.S. News, told The Cancer Letter that this was a huge "screw-up." The hospital systematically misclassified emergency patients, which led to the exclusion of nearly 40 percent of admissions, Paul Goldberg, The Cancer Letter’s editor, reported. He said the error was discovered in 2009, but no way could be found to correct it. "Since U.S. News averages data over three years, the results of the M.D. Anderson top rating by the magazine released July 16 are still partially based on tainted data," he wrote.
Is “error” actually the right word to use?
Oh, you mean we’re also supposed to count the ones that might make us appear less successful. Oops, my bad. Sounds familisr.
I can’t find the source, so that’s a little embarrassing, but I remember that there was one study into why the French rates of heart disease were so low (despite more red meat and wine in their diets) that concluded that the reason was that the French were only counting people treated in hospital – if you dropped dead from a heart attack or died in the ambulance, then it didn’t count as heart disease. This would improve the figures, yes.
Oh, this is going to be a big story if the media decide to start digging. The only reason I’m not sure it’ll happen is that mainstream journalism has been a massive part of the problem here, especially from TIME, CNN, and US News & World Report.
The evidence that the whole MD Anderson-media circus is corrupt is US News bullshit excuse that “no way could be found to correct” the erroneous rankings. Of course there is a way: re-analyse the data with the bias corrected. The patient records will still exist, so correct data should be retrievable if MD Anderson and US News were interested in doing so. And even if US News decided it was too expensive/time-consuming to resample the data in time for this year’s report, they could have pulled MD Anderson from the rankings altogether on the grounds that the ranking was not reliable. To go ahead and publish a list with MD Anderson on top knowing that the ranking data was fudged is completely unforgivable.
Oooh, it gets better. The US News and World Report still has MD Anderson on top of its rankings with no disclaimer attached, and a search of US News website for “MD Anderson” brings up not a single link to this story. The bastards went ahead and published shonky data, knowing it was shonky, and they have been doing their best to hide it from their readers.
I bet MD Anderson pays big money for ads in US News & World Report.
Srsly, PZ? You’re tarring the thousands of cancer researchers at MD Anderson with one brush? I know these people, I work with them regularly. They do some of the best cancer research in the US, along with Memorial Sloan and Fred Hutch. They put in thankless long hours for little pay. You’re making it sound like the entire institution are scumbags.
This is just marketing BS to jockey for some little squiggle on their ads. Don’t confuse the PR department for the whole institution.
marksheffield@6- PZ is most certainly not tarring everyone with one brush. How intellectually dishonest of you to set up that straw man. There is a great likelihood that SOME people at MD Anderson have engaged in fraud. If you care about that institution and the people there, you should hope that those responsible are held accountable for the sake of the institution (and the scientific and medical establishments in general).
There are also dedicated hardworking people at Sloan Kettering & MGH and a thousand other institutions. Isn’t it unfair to them if MDA has finagled the stats to unjustly exaggerate its own achievements?
This seems to be a recurring theme in Cancer treatment, most of the treatment centers that boast above average success rates, get those rates by refusing care to people who wont be considered a success, sometimes even kicking people out of the program if they aren’t showing enough improvement.
I’m trying to find the “tarring” in the 27 words that PZ wrote about MD Anderson.
Ah, tainted statistics. Gotta love ’em.
Of course, it’s a game everyone plays. I love those radio ads for Memorial-Sloan Kettering, about how patients who are treated there first often do better than patients treated at other hospitals. Guess what, patients treated at a magnet center first:
a) Are generally still well enough to travel, rather than already desperately ill.
b) Probably are wealthier and/or have private insurance and therefore
c) Less likely to have interruptions in treatment
d) On average, diagnosed earlier
e) Might have fewer co-morbidities (have to check that one.)
But that’s a radio advertisement. Ads are supposed to slant things. Releasing screwy data for official comparison statistics, yeah, that’s different.
How about they leave the institution out of the listings until they can find a way to correct the “error”, with of course a prominent statement about why they’re being left out. Something like this: “MD Anderson has been left off this list due to their claimed results being inflated by them via either very sloppy recording-keeping or outright fraud.”
What you have not considered is the impact on people like me. My wife died of metastatic breast cancer two years ago and was treated at a local hospital by a top rated (by NJ Magazine) oncologist. She lived 18 months from the onset of symptoms and from what I gather talking to an oncologist from Sloan-Kettering her actual expectancy was 6-8 months. So intellectually I know she received the best care possible and did far better than expected with an excellent quality of life until the last few days but STILL when I see the ads from MD Anderson, Cancer Centers of America, etc. I wonder if she made a mistake and should have gone to a “better” facility.
Seeing how the data was fudged does in a perverse way make me feel more comfortable with the decision she made.
The classic rock station I listen to here in the Adirondacks has been broadcasting an ad for some cancer treatment center (Roslyn something). Listening to the ad, you would think they had some secret knowledge that they use to treat patients. I cringe whenever I hear it.
@chrislawson
To go ahead and publish a list … knowing that the ranking data was fudged is
completely unforgivablewhat they do and what makes them a profitable business.Fixed that for you.
This is SOP for USNews. They’re an advertising medium masquerading as a news outlet.
This comes on the heels of reports of one oncologist in Michigan bilking patients out of quality care and Medicare out of millions of dollars, and not too far back in time Orac was writing about Cancer Treatment Centers of America, and their questionable practices regarding patient outcome statistics.
(for those who do not follow Orac : http://scienceblogs.com/insolence/2013/03/07/the-cancer-treatment-centers-of-america-cherry-picked/ )
I’m starting to lose faith in those of my fellow Americans to run a medical business entirely.
Perhaps it’s time to start taking some of the profit motive out of care delivery ?
That 40% figure sounds rather large. How come the other hospitals didn’t notice? Maybe they are running the same scam but have their exclusions set at a slightly lower level (and decided not to say anything about Houston lest their own scam be revealed).
As an Australian, reading all your posts about different cancer centres advertising on tv and radio sounds very weird.
It’s literally a foreign concept.
As a UKian, I can only echo RobertL’s comment — the very concept takes my breath away (not in any good sense).
Unfortunately my country’s current maladministration is taking us ever further in the direction of the US approach to healthcare. They want to use their five years of dubious power to enact a slash-and-burn approach to the things we’ve held dear for so long, trusting that no subsequent mainstream political party will have the political will (or tax revenue) to reverse these foul and divisive effects upon society. The rich will get richer and the poor will get poorer.
I’d like to think we could do something about it, but the more you look at any group of politicians the more you realise that politicians aren’t the answer. Even hanging the fuckers from lampposts would only provide temporary relief.
@bionichips:
I know this was not the main thrust of your post, but you have my condolences. I am sorry this happened, and I am glad they were able to maintain a good quality of life for your wife, and that you both got an extra 12 months.
For the time before it was noticed, yes. Afterwards, if nothing is done about it- even a simple announcement that the statistics aren’t accurate- it’s lies and possibly fraud.
janewhite, magnet centres also tend to attract more difficult cases and cases that have failed to respond at other centres first, to there is often a bias *against* top quality care. The solution is to be careful with your stats and to collect as many confounders as possible (e.g. socioeconomic status, age, co-morbidities, type and stage of cancer, etc.) and analyse appropriately. I have, of course, zero faith in US News’ capacity (or interest) in doing so.
marksheffield: PZ didn’t tar anyone but the MD Anderson administration team and US News for fudging outcome data.
bionichips,
Sorry for your loss, but you actually don’t know how good the care your wife received was other than at the palliative end (which IMHO is even more important). That she did better than the median doesn’t mean it was because her treatment was better. I’m not saying this to undermine your opinion of her oncology team, I’m saying this to add to your observation that these kind of ranking systems play on people’s need for certainty in the face of hugely stressful, highly variable diseases. Even if you know about the flaws in their statistical methods, your emotional response is exactly what the bastards were hoping to achieve with their marketing spin. The process is designed to make people with terrible diseases (and their loved ones) seek out these centres in desperation.
This is this the inevitable result of market driven medicine. The magazine articles accompanied by ads for the cancer centres reminds me of the free* magazines you get in the health food / natural food stores where an article about some miracle supplement is paired with an ad for that supplement.
*Overpriced in my opinion.
I agree with your general point, PZ, but yes, we should call it an error for now. Until there’s more definitive proof than the above to show misdeeds, while I am certain they were there and were caused by cash, we should stick to language that allows the situation to resolve IF (bigger if than the font here will display) they should find an actual, honest error. As our noble hero, Rep. Gohmert, would say, you’re casting aspersions on their asparagus.
The error has been publicly admit and is in print. Take your concern and shove it where the sun don’t shine.
An honest hospital administration would insist on being taken out of the rankings or applying a 30-40% reduction to its ratings.
Where there is profit to be made, there are parasites, scum-suckers, thieves, and fraudsters.
@chrislawson: You’re right. People who are treated at magnet centers as a last resort are going to have pretty bad outcomes on average, even if many do far better than expected. But what I was criticizing was the commercials bragging about the great outcomes of patients treated at a magnet center FIRST.
In my sector of the education industry, we know all about how difficult it is to compare graduation rates fairly between institutions, and I’m sure cancer cure rates/survival times are far harder to compare.
janewhite: sorry, I didn’t mean to come across as correcting you. I think I’ve had a bad week (it’s election week in Oz and the media is full of so much uncorrected BS I’m almost ready to explode) and I think it’s been making me speak even more bluntly than usual. Mind you, I don’t think it’s any easier to compare graduation outcomes between schools/colleges and health outcomes between hospitals. They’re both very difficult to measure in a reliable way, prone to manipulation (and not just in market-oriented institutions), and susceptible to stupid but popular political pseudo-solutions like publishing league tables.
The topic of cancer research interests me enough to risk noob embarrassment in discussing it. I was reading this article, and although there are a suite of factors that contribute to cancer it gives me a glimmer of hope that there may be a silver bullet somewhere in the development of cancer to stopping it. http://phys.org/news151840958.html
The title is a thought provoking question “Why do the majority of people never get cancer?”
Interesting research has largely ignored the question why the majority of people don’t get cancer because they focused solely on genetics. They think they know the answer they are genetically predisposed to cancer so they get cancer. So the focus of research becomes too focused on families with cancer histories. Sure but what traits do people inherit that makes them cancer resistant?
There is a silver bullet there I know it for a very smart scientist to stop cancer progression. We just have to ask the right questions. There will be a number of dead ends but we haven’t tried enough paths.
Klein’s talking tumor microenvironment as a modulator of malignant transformation and tumor susceptibility.
Maybe it’s the emeritus thing, but in that interview, Klein strikes me as unaware a lot of recent cancer research, which focuses on tumor microenvironment and a lot of things that he claims cancer researchers ignore. I mean, seriously, as a cancer surgeon and researcher myself, I was facepalming as I read the linked article. I mean, holy crap, it’s not as though we haven’t been studying tumor microenvironment as related to immune surveillance of normal tissue for transformed cells for, oh, maybe 50 years or more. Or as related to tumor angiogenesis for the last 25 years or so. Or direct interactions between stroma and tumor cells as regulators of neoplastic transformation for at least 20 years. Perhaps I’ll go look up the PNAS article; it could be that Klein’s argument is not well presented in that news article.