Nov 05 2012

The Dodo Bird Verdict

Everybody has won, and all must have prizes.

First it marked out a race-course, in a sort of circle, (`the exact shape doesn’t matter,’ it said,) and then all the party were placed along the course, here and there. There was no `One, two, three, and away,’ but they began running when they liked, and left off when they liked, so that it was not easy to know when the race was over. However, when they had been running half an hour or so, and were quite dry again, the Dodo suddenly called out `The race is over!’ and they all crowded round it, panting, and asking, `But who has won?’

This question the Dodo could not answer without a great deal of thought, and it sat for a long time with one finger pressed upon its forehead (the position in which you usually see Shakespeare, in the pictures of him), while the rest waited in silence. At last the Dodo said, `EVERYBODY has won, and all must have prizes.’

`But who is to give the prizes?’ quite a chorus of voices asked.

`Why, SHE, of course,’ said the Dodo, pointing to Alice with one finger; and the whole party at once crowded round her, calling out in a confused way, `Prizes! Prizes!’

(Alice in Wonderland, Lewis Caroll)

The Dodo Bird Verdict postulates that different orientations of therapy (cognitive-behavioral, rational-emotive, etc.) don’t have significantly different outcomes. They’re all winners! They all get prizes! That is to say, it’s less a matter of which kind of therapist you go to…and more a matter of what kind of relationship you develop with them. Are you comfortable disclosing? Do you feel that they’re invested in your success? Do you feel like you’re being patronized or led in circles? Strong therapist-client bonds matter.

[Sidenote: the studies and analyses I'm going to run through didn't examine therapeutic outcomes in children or in those with psychotic features as part of their disorder.]

[Second sidenote: psychotic and psychosis are specific scientific terminology, and not adjectives for people who do weird and/or disagreeable things. Also, having psychotic features does not necessarily equal having schizophrenia. /rant]


Luborsky, Rosenthal, et al. do a really fascinating meta-analysis of seventeen meta-analyses (a Meta-Meta-Analysis?). In essence, when research corrects for the allegiance of the therapist (who may prefer on type of treatment to another), there are small, non-significant differences between treatments across patient types. Really, I’d suggest reading that study in its entirety (it’s freely available!).

The point is one I’ve tried to make before, albeit with fewer citations: If you are able to go to therapy, how you feel about your therapist matters. Ask questions! Conversely, if you feel like your counselor just isn’t getting it, it’s not necessarily because therapy doesn’t work. It’s not even necessarily because your therapist’s orientation is wrong. You may not find them to be empathetic, or to fit your personality. In fact, one of the tangets of the Luborsky article presents evidence that while across patients, outcomes in different therapeutic orientations are different in non-significant ways, there is research to suggest that pairing subgroups, such as temperament and personality types with specific types of therapy may account for slightly better outcomes.

That is to say, if you want to be given concrete direction in your life, you might develop a better client-therapist bond in a type of therapy where the counselor is viewed as the expert or teacher for the client. As someone who studies psych services and likes to draw their own conclusions, I prefer therapy with a ‘team’ structure; the therapist and I are both working together to fix things. Current research suggests that if you and I switched therapists, we’d both do better than if we didn’t attend therapy at all, but we might be more frustrated and less fully involved in the process.

So. Go read that article!


  1. 1
    Christoph Zurnieden

    (a Meta-Meta-Analysis?)

    And herein lies the problem: the method used is questionable.
    You cannot use the raw metastudies, they might overlap, differ slightly but signifcantly in the question/method, might be biasedhave still undetected errors, and so on. So you need to take the individual studies listed there and do your own metastudy with these. That is not what they did, at least not completely and that might have caused undetected errors.

    Doing metastudies is a very complicated thing and needs a good statistician, they used an old textbook: the good ol’ Cohen from 1977. It is not a bad textbook, far from it, but a textbook does not and cannot replace a statistician.

    On a sidenote: they used only the studies that compared a therapy with a different therapy. They might have had a good reason to do so but this is the reason they gave (from the article, Footnote 3, page 17):

    Consider a study comparing T1 vs. Control1 that finds d = .80 and a study comparing T2 vs. Control2 that finds d = .30. We conclude T1 is better than T2 because a d of .80 is larger than a d of .30. However, if the study of T2 had employed a much sicker population of patients, the smaller d is not due to a difference between treatments but to a difference between clienteles. A head-to-head comparison of T1 vs. T2 for a sample of patients for which both T1 and T2 would be appropriate might find no difference at all.

    That’s not how you do it, really not. If you want to compare both therapies you have to include the control groups. There are many well understood and moreso well known statistical techniques too choose from. I don’t have the Cohen at hand, but I’m pretty sure it gets treated there, too.

    The study itself might nevertheless be valid and there is a good chance it is, but the method used definitely isn’t.


  2. 2
    Miriam, Professional Fun-Ruiner

    If you want to compare both therapies you have to include the control groups.

    …not really? If you’re only comparing therapies to each other, you don’t need to include the control groups. But if you’re trying to make claims about the overall effectiveness of a given therapy orientation, then you need to compare it to a control group.

  3. 3
    Christoph Zurnieden

    …not really?

    Yes, really.
    Both therapies might differ from each other but not from the control group, or more blunt: both therapies might differ but only in the ways they do not work or both therapies do not differ but only one works.
    Moreso: including the control groups allows for normalizing the effectivity of the individual therapies to get some values (with defined errors) which then can be compared directly. That is one of the reasons a meta-metastudy makes no sense, only metastudies are meaningful. (with an exception if all of the metastudies that are surveyed have been normalized in the very same way. Not impossible, yes, but not very likely.)

    A metastudy differs from a “normal” study only insofar that you have several datasets instead of only one and the method of sampling that data might, or better will be different for each different dataset. It is obvious that the very first thing you have to do before you are able to compare the datasets is to normalize them. You cannot do it without access to the full data, which not everyone is able or willing to give.
    So you start your metastudy with a large pile of data of different quality, resolution and method of collection and begin to normalize it to be able to compare them. That is the main piece of the whole thing, 95% of the work, the part where you can show your brilliance and knowledge. The rest is just statistics 101.
    And now tell me exactly how they normalized their data? You can’t because they didn’t do it at all. They dumped half of the data (the control groups), compared apples to oranges and called both fruits round in their conclusion because a lot of people said so.

    If the health insurance companies have good reasons too assume (this paper is one “good reason” for them) that all therapies have the same effectivity they will pay for the cheapest only or they pay all therapist the same amount. But only if the therapies can be shown to work! So you really need the control groups, you can’t drop them. The health insurance companies take every chance to avoid spending.

    So, if a study needs a lot of work and knowledge to do and you are not willing to invest the necessary work and/or able to do it, then, for the sake of the wellbeing of the people, just don’t do it. Your tenure is not worth it.

    The irony of that all: I think their hypothesis holds, they are right and with the correct method they even would have had a small but not that small chance to get published in a “bigger” journal. Probably not in Nature/Science/Lancet but who knows. That would have brought the information to a much wider audience (e.g.: an article in Nature has a good chance to make the news) including the people who are in need of it. Also: publishing in that kind of journals looks really good on the CV.
    Aaaand you can brag about it at the parties.

    Well, I think having too much time at my hand does not too much good to the clarity and shortness of my arguments, so I’ll better stop here ;-)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite="" class=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>