A couple of weeks ago, Ophelia highlighted a proposed (since cancelled) Skeptics in the Pub talk by someone who is “skeptical” that any inequalities still exist that disadvantage women. From the description of the talk:
Leeds psychologist Dr Gijsbert Stoet finds no evidence that women under-perform through internalising false stereotypes, a recent major review reveals no sex-discrimination in academia, and ground-breaking field research shows that it is actually in favour of women in recruitment; so why is it women tend not to ‘get to the top’?
The Stoet paper on stereotype threat is available by request from his university, so I read it for myself. Who would like to guess whether it shows “no evidence” of stereotype threat? Who would like to guess whether the existence of stereotype threat is even the research question the paper addresses?
Right. No one’s going to take the fool’s bet when I phrase it that way. Good, because that’s a lousy representation of the paper, which is titled, “Can Stereotype Threat Explain the Gender Gap in Mathematics Performance and Achievement?”
Those familiar with discussion of social science research on inequalities will recognize this kind of question as a “magic button” question. Is stereotype threat the one overwhelming thing that causes a gender gap in mathematics? Um, does any social phenomenon have just one overwhelming cause? Generally not, but that doesn’t stop Stoet, and his coauthor David C. Geary, from asking exactly that. From the conclusion of the paper:
This article reviewed evidence for the stereotype threat explanation of gender differences in performance, favoring boys and men, on difficult mathematics tests (gender differences are not typically found on comparatively easy tests; Penner, 2003). The question was whether the published research provides strong and stable evidence for the stereotype threat hypothesis as the primary causal explanation of this gender difference.
So the question of the paper was not “Does stereotype threat exist?” or “Does stereotype threat contribute to gender differences on difficult mathematical tests?” If it had been, the answer provided by the paper would have been an unequivocal “Yes.” The authors excluded every study that did not use a male control group or used an adjustment to math scores, and they still found that that the existence of stereotype threat was supported in 30% of situations studied. To quote the paper again:
The available evidence suggests some women’s performance on mathematics tests can sometimes be negatively influenced by an implicit or explicit questioning of their mathematical competence, but the effect is not as robust as many seem to assume. This is in and of itself not a scientific problem, it simply means that we do not yet fully understand the intrapersonal (e.g., degree of identification with mathematics) and social mechanisms that produce the gender by threat interactions when they are found.
This, of course, very different from what the description of the Skeptics in the Pub talk says. There is evidence of stereotype threat that has been replicated reliably, if not universally. For a social phenomenon, that’s pretty good.
So where did our Skeptics in the Pub speaker get his interpretation of the paper? Well, as it turns out, a lot of papers, particularly on subjects that are controversial enough to garner media attention, have press releases issued by a researcher’s institution. This one is no exception. And what does this press release say?
“The stereotype theory really was adopted by psychologists and policy makers around the world as the final word, with the idea that eliminating the stereotype could eliminate the gender gap,” said David Geary, Curators Professor of Psychological Sciences in the MU College of Arts and Science. “However, even with many programs established to address the issue, the problem continued. We now believe the wrong problem is being addressed.”
In the study, Geary and Giljsbert Stoet, from the University of Leeds in the United Kingdom, examined 20 influential replications of the original stereotype theory study. The researchers found that many subsequent studies had serious scientific flaws, including a lack of a male control group and improperly applied statistical techniques.
“We were surprised the researchers did not subject males to the same experimental manipulations as female participants,” Geary said. “It is reasonable to think that men also would not do well if told ‘men normally do worse on this test’ right before they take the test. When we adjusted the findings based on this and other statistical factors, we found little to no significant stereotype theory effect.”
The researchers believe that basing interventions on the stereotype threat is actually doing more harm than good, as vital resources are being dedicated to a problem that does not exist.
So, the paper shows an effect even after removing the studies without male control groups, but it’s important to mention those studies don’t have male control groups–and that being told men don’t perform well at a math test could, if it ever happened, possibly affect men’s performance. The paper says that, to the extent stereotype threat is presented as the only problem affecting women’s performance, there is too much attention paid to stereotype threat, but the press release says paying any attention to it at all is a problem. The paper finds replication of the effect, but the press release calls it “a problem that does not exist”.
It’s a sad state of affairs when a press release and a paper are this far off on their conclusions, but it does, once again, demonstrate how important it is to read the one of them that is peer-reviewed.
Stoet, G., & Geary, D. C. (2012). Can Stereotype Threat Explain the Gender Gap in Mathematics Performance and Achievement? Review of General Psychology, 16 (1), 93-102 DOI: 10.1037/a0026617