Biased Tests

[cn: Bayesian math]

Suppose that I create a test to measure suitability for a particular job. I give this test to a bunch of people, and I find that women on average perform more poorly. Does this mean that women are less suitable for the job, or does it mean that my test is biased against women?

Psychologists do this all the time. They create new tests to measure new things, and then they give the tests to a variety of different groups to observe average differences. So they have a standard statistical procedure to assess whether these tests are biased.

But I recently learned that the standard procedure is mathematically flawed. In fact, rather than producing an unbiased test, the standard procedure practically guarantees a biased test. This is an issue that causes much distress among psychometricians such as Roger Millsap.

Following Millsap, I will describe the standard method for assessing test bias, sketch a proof that it must fail, and discuss some of the consequences.

[Read more…]

Paper: Attack of the psychometricians

Suppose that you want to demonstrate that baby boomers are more narcissistic than other generations, or that women are more agreeable and neurotic than men, or that people of different races have different amounts of intelligence. How do psychologists do that? Can they in fact do that?

Typically, the method is to come up with a bunch of questions that superficially appear to measure the intended characteristic. Then the questions are “validated”, for example, by making sure the questions all correlate with one another. Once the questionnaire is declared valid, psychologists can then measure a variety of different groups and make far-reaching claims about how our current political/social situation was caused all along by the thing that they happen to study.

If you find this methodology questionable, but aren’t sure exactly what went wrong, you might be interested in hearing about psychometrics, the field concerned with psychological measurement. According to psychometricians, part of the problem is that psychologists are failing to follow best practices. That is the subject of this paper:

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440.

[Read more…]

Paper: Overbooking flights

In recent news (cn: autoplay), a United Airlines passenger was unwillingly dragged off a plane to make seats for employees. The incident was partially blamed on the common practice of airlines to deliberately overbook flights in order to make up for all the no-show passengers.

While I won’t discuss the particulars of this incident, I am willing to do something that most journalists are not: read relevant academic literature. I just picked one paper that appeared to have a sufficiently broad perspective:

Marvin Rothstein, (1985) OR Forum – OR and the Airline Overbooking Problem. Operations Research 33(2):237-248.

The basic problem is that many people who buy airplane tickets don’t show up. In 1961, the Civil Aeronautics Board (CAB) reported that 1 out of 11 ticket sales were no-shows. These numbers are about the same today, with 7-8% no-shows. The airlines could create wait lists to fill the empty seats, but it would be impossible to contact the wait-listed customers in a timely fashion unless they were already present at the gate.

[Read more…]

Paper: Tops and Bottoms

cn: This post discusses m/m sex and genitals. I don’t joke about sex, so if you’re expecting sex jokes stop expecting that.

When I was reading that article about gay loneliness, I followed a reference to “A Longitudinal, Mixed Methods Study of Sexual Position Identity, Behavior, and Fantasies Among Young Sexual Minority Men” by Pachankis et al.

“Sexual position identity” refers to “top” or “bottom” or “versatile”. I would guess that most of my readers are already familiar with these terms, but I don’t want to be presumptive so I’ll just spell it out. The identity terms refer to sex positions in anal sex, with “top” being the insertive position, “bottom” being the receptive position, and “versatile” meaning no strong preference either way.

I will be upfront about my prejudices. These identity labels don’t make much sense to me. If people prefer one sex position over another that’s fine but an identity labels aren’t really useful unless they convey some information that a lot of people need to know. The only people who really need to know are sexual partners, or I suppose potential sexual partners. And we’re talking specifically anal sex, which contrary to stereotypes is not actually the most common sexual practice between men. So, sex position identity labels might make sense if you have a lot of sexual partners, but not otherwise. Given the prevalence of sex position identity labels, I strongly suspect that they are fulfilling some other function, like being a vehicle for stereotypes.

Yes, there are top and bottom stereotypes. Bottoms are supposed to be more submissive and feminine. I don’t understand it. [Read more…]

Kochen-Specker Theorem explained

I previously explained Bell’s Theorem, which is a “no go” theorem of quantum mechanics. In brief, Bell’s Theorem proved in 1964 that any hidden variable interpretation of quantum mechanics must be nonlocal.

Of course, you may be thinking, maybe the world just is nonlocal, and that hidden information is being passed around faster than light. Unfortunately, there’s another major theorem which makes hidden variable theories even more unpalatable. In 1966-1967, the Kochen-Specker Theorem proved that any hidden variable interpretation must be contextual.

To understand the meaning of “contextual”, suppose we have a quantum cat, and the cat has many possible states. It could be awake or asleep. It could be happy or unhappy. Or the cat could be none of those things because it is dead. Now suppose there are two possible measurements, which answer the following questions:

(1) Is the cat awake, asleep, or dead?
(2) Is the cat happy, unhappy, or dead?

This is a quantum cat, so you can only choose one of the two measurements. However, even if you can’t make both measurements experimentally, you might reasonably expect that the outcomes of the two measurements are related to each other.  Specifically, if measurement (1) would find a dead cat, then so would measurement (2), and vice versa. This assumption is called non-contextuality. This cannot be true of hidden variable interpretations of quantum mechanics! Such theories must be contextual.

Figure 1: ambiguous catFig. 1: Cat of ambiguous state.  Credit: Visentico / Sento

[Read more…]

Made in Criticalland

This is a repost of an article I wrote in 2014.  Relevant to my recent review of the Sokal paper.  Note that the blog Scientia Salon is now defunct.

Massimo Pigliucci started a new blog Scientia Salon, which is already bearing fruits.  I enjoyed this essay by Alan Sokal (yes, that Sokal) about academic postmodernists and extreme social constructivists.  In the 80s and 90s there were many such academics claiming that science was entirely based on prejudices.  Interestingly, Sokal claims that they have now backed off from the most extreme views, particularly because they were upset at the way the Bush regime used postmodernism to justify its anti-science policies.

Sokal’s primary citation for this is “Why has critique run out of steam? From matters of fact to matters of concern” by sociologist of science Bruno Latour in 2004.  I thought it was worth a read. [Read more…]

Why isn’t homosexuality (or religion) a mental disorder?

In a comment discussion last month, we touched on the question of whether religion could ever be considered a mental disorder. This is a common idea among atheists, sometimes expressed as a joke, or sometimes claimed seriously. I am not mentally ill, so I would defer to other people to explain why it is wrong to compare religion and mental illness even as a joke. Here I will ignore the jokes and consider only the serious question: Why isn’t religion a mental disorder?

According to the DSM-5,

A mental disorder is a syndrome characterized by clinically significant disturbance in an individual’ s cognition, emotion regulation, or behavior that reflects a dysfunction in the psychological, biological, or developmental processes underlying mental functioning. Mental disorders are usually associated with significant distress or disability in social, occupational or other important activities. An expectable or culturally approved response to a common stressor or loss, such as the death of a loved one, is not a mental disorder. Socially deviant behavior (e.g., political, religious, or sexual) and conflicts that are primarily between the individual and society are not mental disorders unless the deviance or conflict results from a dysfunction in the individual, as described above. [emphasis mine]

There you go. Religious behavior isn’t a mental disorder because the DSM-5, an authoritative document, says so. However, you could be forgiven for not taking the DSM’s word for it. Let’s dig deeper.

Look at what else has been excluded from mental disorders: socially deviant sexual behavior. This exclusion arises from a famous controversy, which led to the declassification of homosexuality as a mental disorder in the DSM in 1973. And until 1987, homosexuality remained as a mental disorder (“Sexual Orientation Disturbance” and later “Ego-dystonic Homosexuality”) as long as the patient was distressed about their orientation. The architect of these decisions was psychiatrist Robert Spitzer. I believe that Spitzer himself offers the best insight into the definition of mental disorders.

[Read more…]