A couple of weeks ago, Ophelia highlighted a proposed (since cancelled) Skeptics in the Pub talk by someone who is “skeptical” that any inequalities still exist that disadvantage women. From the description of the talk:
Leeds psychologist Dr Gijsbert Stoet finds no evidence that women under-perform through internalising false stereotypes, a recent major review reveals no sex-discrimination in academia, and ground-breaking field research shows that it is actually in favour of women in recruitment; so why is it women tend not to ‘get to the top’?
The Stoet paper on stereotype threat is available by request from his university, so I read it for myself. Who would like to guess whether it shows “no evidence” of stereotype threat? Who would like to guess whether the existence of stereotype threat is even the research question the paper addresses?
Those familiar with discussion of social science research on inequalities will recognize this kind of question as a “magic button” question. Is stereotype threat the one overwhelming thing that causes a gender gap in mathematics? Um, does any social phenomenon have just one overwhelming cause? Generally not, but that doesn’t stop Stoet, and his coauthor David C. Geary, from asking exactly that. From the conclusion of the paper:
This article reviewed evidence for the stereotype threat explanation of gender differences in performance, favoring boys and men, on difficult mathematics tests (gender differences are not typically found on comparatively easy tests; Penner, 2003). The question was whether the published research provides strong and stable evidence for the stereotype threat hypothesis as the primary causal explanation of this gender difference.
So the question of the paper was not “Does stereotype threat exist?” or “Does stereotype threat contribute to gender differences on difficult mathematical tests?” If it had been, the answer provided by the paper would have been an unequivocal “Yes.” The authors excluded every study that did not use a male control group or used an adjustment to math scores, and they still found that that the existence of stereotype threat was supported in 30% of situations studied. To quote the paper again:
The available evidence suggests some women’s performance on mathematics tests can sometimes be negatively influenced by an implicit or explicit questioning of their mathematical competence, but the effect is not as robust as many seem to assume. This is in and of itself not a scientific problem, it simply means that we do not yet fully understand the intrapersonal (e.g., degree of identification with mathematics) and social mechanisms that produce the gender by threat interactions when they are found.
This, of course, very different from what the description of the Skeptics in the Pub talk says. There is evidence of stereotype threat that has been replicated reliably, if not universally. For a social phenomenon, that’s pretty good.
So where did our Skeptics in the Pub speaker get his interpretation of the paper? Well, as it turns out, a lot of papers, particularly on subjects that are controversial enough to garner media attention, have press releases issued by a researcher’s institution. This one is no exception. And what does this press release say?
“The stereotype theory really was adopted by psychologists and policy makers around the world as the final word, with the idea that eliminating the stereotype could eliminate the gender gap,” said David Geary, Curators Professor of Psychological Sciences in the MU College of Arts and Science. “However, even with many programs established to address the issue, the problem continued. We now believe the wrong problem is being addressed.”
In the study, Geary and Giljsbert Stoet, from the University of Leeds in the United Kingdom, examined 20 influential replications of the original stereotype theory study. The researchers found that many subsequent studies had serious scientific flaws, including a lack of a male control group and improperly applied statistical techniques.
“We were surprised the researchers did not subject males to the same experimental manipulations as female participants,” Geary said. “It is reasonable to think that men also would not do well if told ‘men normally do worse on this test’ right before they take the test. When we adjusted the findings based on this and other statistical factors, we found little to no significant stereotype theory effect.”
The researchers believe that basing interventions on the stereotype threat is actually doing more harm than good, as vital resources are being dedicated to a problem that does not exist.
So, the paper shows an effect even after removing the studies without male control groups, but it’s important to mention those studies don’t have male control groups–and that being told men don’t perform well at a math test could, if it ever happened, possibly affect men’s performance. The paper says that, to the extent stereotype threat is presented as the only problem affecting women’s performance, there is too much attention paid to stereotype threat, but the press release says paying any attention to it at all is a problem. The paper finds replication of the effect, but the press release calls it “a problem that does not exist”.
It’s a sad state of affairs when a press release and a paper are this far off on their conclusions, but it does, once again, demonstrate how important it is to read the one of them that is peer-reviewed.
Citations
Stoet, G., & Geary, D. C. (2012). Can Stereotype Threat Explain the Gender Gap in Mathematics Performance and Achievement? Review of General Psychology, 16 (1), 93-102 DOI: 10.1037/a0026617
The weird thing about the Skeptics in the Pub talk was that the guy they had scheduled wasn’t even involved in the research. He is an author of a pretty badly researched and sourced book called ‘The Women Racket’.
I wrote about it further over at The Heresy Club: http://heresyclub.com/2012/07/leeds-skeptics-i-think-youve-made-a-big-mistake/
Nothing really adds up when it comes to the talk they had planned.
And then there will be the next level of distortion when Steven Moxon presents what he takes to be the findings of Geary and Stoet – which is one reason the more irritated members of Leeds SITP suggested at least inviting Stoet instead of Moxon.
Oh – ! Happens again. I was typing while Hayley was posting. Could have saved myself the trouble.
Thank you for digging up the original paper for this. I’m almost at the point where I just don’t believe anything in a press release.
Stephanie, do check out this paper – recommended by a social psychologist friend – on the history of stereotype threat and how it’s been misinterpreted over the years:
http://www.asc.upenn.edu/usr/ogandy/C45405%20resources/Sackett%20et%20al%20stereotype%20threat.pdf
You know that press releases about research are typically written by administrators and PR people without a clue about the given field of research, right? A good number of them just takes a few important-sounding words and soundbites and connects them up without even knowing if the resulting sentence is semantically viable, much less if it actually means what the source is saying.
This press release is a prime example why science PR shouldn’t be left to the PR people without substantially involving actual scientist.
Did I understand their idea of a control group properly? It sounds like they are equating telling a man that women perform better in a particular test once right before the test to women being told they are inferior to men (in math and science) all their lives? I see a flaw there. But they can’t actually mean that, right?
Maude, in both cases, the subjects would receive the message about their own gender being bad at taking this test. The control would be the presence of the stereotype, which does exist for women and does not exist for men, at least not for math. The idea is that the stereotype must be present in order for the gender-specific message to affect performance.
Ysanne, you did notice that most of the press release that I quoted is direct quotations from one of the researchers, yes?
I do, and it looks somewhat oversimplified and may well be taken out of context.
Seriously, I see this “we have to put some impressive research result on our web site!” panic first hand on a daily basis, and usually the researchers (or conference organisers or whatever) only find out that a media release has been put up when they happen to browse the department/uni home page. It’s a dead embarrassing thing to realise that “students performed complicated calculus equations” has been the headline for over a week… it gets worse when it turns out that this was taken verbatim from an official uni newsletter that goes to prospective international students, along with some “quotes” by people who were never contacted.
So: Yes crappy press release, yes possibly too bold wording by researcher, but no way sure how the flat-out denial of stereotype threat got in there. It’s definitely not what Geary wrote.
The weird thing for me is that *even if* this paper did conclude that stereotype threat doesn’t exist/have an effect, stereotype threat has been supported by so many studies that it would need many many more studies finding no effect to make it even remotely reasonable to declare something as grandiose as stereotype threat being over or gone or never here.
Ah, yes, press releases.
That’s how I learned over the years to go and check if tehre was something spectacular in the press. Most times you’re going to be disappointed, angry or both.
But I’m wondering a bit about the control-group question (I remember from the “Gender Delusion” that you actually can test this with a male control in some cases, but how are you going to control for a phenomenon that is present since we’re born?
I mean, you can’t just change the message to “men are bad at maths” unless you raise a group of men in a Trueman-show scenario.
What Giliell said. It’s impossible to find a control group that isn’t in some way conditioned to common social/cultural memes and tropes. I do think that looking at stereotype threat is one angle on a much bigger problem, and that bigger problem would be easier to test for.
Three groups, randomly chosen. One group is told that the average mark on the test is 80. Another group is told that the average mark on the test is 50. A third group is told nothing. See how the test results compare. Do a second test, mix up the groupings, and see how the results compare to the original.
It’s not exactly the same..with stereotype threat you’re talking about internal expectations and with that test you’re looking at external expectations, but I would suspect you would get similar results, due to things such as confidence and stress.
One reason why I think one would see a rather big pushback against these concepts is that they directly challenge some pretty big concepts in terms of academics and education. (I.E. with high stakes testing we’re not only testing knowledge, we’re also testing relative stress levels, making the results basically useless if you’re looking to measure knowledge)
I have a feeling that some skeptics somehow think that going against commonly accepted ideas (such as stereotype threat existing) is necessarily “skeptical”.
Not so much.
Jesus. I write these press releases sometimes, and the researcher I write them for would kick my ass if I ever misrepresented his work so badly. Then again, I actually sit down with him to get the quotes for the releases and he looks them over before they go out. Apparently that’s an unusually high standard.
How about we change the term from “stereotype threat” to “fear of being raped from being a competent women threat”?
That would be something you could test on men. Perhaps not ethically, I doubt any IRB would allow a protocol that would raise the fear of being raped in men to the level it is in women.
#12, #13: About the controls: Telling men that men generally suck at the kind of test they’re going to take (or make them aware of their sex) would control for the impact of being aware of one’s sex and/or hearing some negative comment about one’s expected performance right before taking the test.
The point would be to filter out the influences of these events that happen immediately before the test, in order to make the actually existing stereotype the only (known) difference between the male and female groups.
If you like to listen to stories (and I do) NPR did one recently on topic:
http://www.npr.org/2012/07/12/156664337/stereotype-threat-why-women-quit-science-jobs
I also liked the solution the NPR reported suggested; hire more women in science.
It might be just me, but I am not sure about Stephanie’s interpretation here. I thought that the authors did a “sceptical” analysis of a popular theory, and they concluded that the evidence is what they call “weak at best”. They seem to show that many of the studies that claimed to have found the stereotype threat effect use poor methodology and that only few of the papers that do not use poor methodology show it. They also point out all sorts of flaws. Are they not just doing what you expect from a sceptical analysis? Not sure, what do you think? I found a video on youtube of the authors that explains it a bit better than the press release.
Ok, maybe I got it wrong. It is not that easy to follow this kind of stuff!
Christina, I’m not sure which part of “Their study supported the existence of stereotype threat but was reported as though it didn’t” you’re having trouble with. However, since you’re in Leeds, you could ask Stoet for clarification. Do you know him already?
I haven’t met him, it is a small world but still a big city. I know one of the guys that knows him, so will forward my questions about this.
[…] that their review of the literature said that stereotype threat wasn’t a real problem. I read the paper and discovered otherwise. The question of human races as “subspecies” came up again, and I talked to a […]