Say I want to settle the question of who’s more moral, atheists or the religious. I’ve got a lab and a grant and some spare time, (A girl can dream, can’t she?) so I set up the experiment that will solve the question once and for all.

Say I bring a bunch of religious people and a bunch of atheists into my lab. I’ve got two research assistants, one of whom plays as if they’re a participant too. So each time someone, religious or atheist comes into the lab, they sit down next to a stranger (my research confederate), and I call them both into the lab together. They sit down, sign lots and lots of consent forms, and do some silly tasks. None of these tasks matter, they’re just there to distract the participants from realizing what I’m actually paying attention to. At the end of doing all the questionnaires, the confederate stands up, and ‘accidentally’ lets and expensive looking watch fall. Without appearing to notice, they leave the room. I see the watch fall and in a distressed voice, ask the real participant if they don’t mind going to the next room and giving the ‘participant’ his watch back, since I’m so busy entering the data.

Now, the real participant could take the watch, walk into the next room, sprint past the fake participant, and leave with a nice new watch. They are, after all, holding onto a watch, I am looking at my computer and entering data, and the ‘participant’ appeared not to notice they’d dropped a watch.

They could.

But most of them don’t. It doesn’t matter if the participant is religious or an atheist, they tend to pick up the dropped watch, walk into the next room, and give it back to our confederate in this experiment.

Case closed, says I! Religious people and nonreligious people are equally moral! After all, they had equal rates of watch-stealing (that is, none at all).

Not so fast, says you. Practically nobody will steal a watch when you’re just sitting there watching* them! You’re there at your computer, the fake participant is in the next room, and you have their name from participant registration and the consent forms! This is a terrible measure of morality–you have to be fantastically immoral to fail this test! In fact, what you’ve done is determine that nonreligious people and atheists have equally low levels of Horribly Immoral and Brazen Watch Thieves.

In fact, says you (why you’ve come into my laboratory to shout at me, I’m unsure), atheists are more moral! If you made this study more complicated–made it easier to steal the watch without suffering consequences, fewer atheists would steal the watch. You’re wrong, says my religious lab assistant! Fewer religious people would steal the watch!

And all the while, I sit there in puzzlement, because I did this study, right? And I was testing for morality, right? Everybody agrees that stealing a watch is Bad and not stealing a watch is Good.** And my research assistants sit there in outrage, because OBVIOUSLY the [religious/atheists] would be more moral if you made the test harder!

This, dear readers, is the ceiling effect. My bar (or ceiling) for Moral Person is far far too low. Everyone returns the watches, but there’s no way to distinguish between the ones who give the watch back and then glare at puppies on the walk home and the ones who return the watch and wander over to the soup kitchen to volunteer.

Take another, real life example. Jacob says men are better at math than women. Elizabeth says this is clearly false. (Both of them are grievously oversimplifying ‘math’, but we’ll let them get away with it.)

Elizabeth points out that This Math Test (TMT, an official exam given to every high school student in our fictional universe) shows that men and women don’t differ significantly. Therefore, men and women are basically about the same in math ability.

Jacob disagrees. He claims that this test is too easy–that men and women do score the same on the TMT, but that doesn’t mean they have the same ablities–the test is too easy. After all, says he, standardized tests hardly examine the highest possible skill level–they cover basic material. He claims that Elizabeth is just demonstrating the ceiling effect–when you give people a really hard test, men outscore women. Jacob is actually right, but this gap is rapidly shrinking, and men also are overrepresented on the other end–with unusually low math performance.*** (Third section after the abstract, here)

And these ceiling debates play out in a number of parts of psychology research. (And in case you didn’t have enough architecture metaphors in your life, we also have the floor effect.) Here’s a more complicated version of the gender-ceiling issue. You can have sparkling methodology, a huge and representative sample base, but if you’re creating a test with a ceiling problem…you might get entirely unhelpful, or worse, misleading, answers.

This is the best and worst of psychology, for me. That there’s always just a little bit more than the research, always a little bit more to debate and argue and question. Maybe the study is too old, maybe you got a weird subset of the population. Maybe the rats are afraid of the gender you always use for research assistants. Maybe there’s a ceiling. Or a floor.

*sorry, this was unintentional.
…mostly.

**That one girl who stole the watch in order to sell it for medication to save her dying father was dismissed as an outlier. 

***Basically, men have higher variance of performance: they’re some of the best and worst performers. Women have a narrower bell curve of math performance.