Sure, there’s some kerfuffle about Ubisoft’s survey, which crashed out if you said you were a female. That’s stupid. But what’s really annoying, to me is…

It’s another survey that’s going to collect self-selected samples.

A “self-selected sample” is a kind of sampling bias in which you collect responses only from people who are bored enough or interested enough to take the survey. Taking that into account, you can see how a survey that collects a self-selected sample is going to generally be worthless. Like the NRA’s survey which showed that an almost overwhelming majority 80%(!) of police think background checks will have no effect on gun crime. A correct title would be: “80% of internet users who cared enough to take our survey and who claimed that they were police say they think background checks will have no effect on gun crime.” Somehow that latter title sounds a lot less conclusive, doesn’t it?

One of my favorite stories of sampling bias was one company that tried to do an online marketing survey, and offered a nice laptop as a prize to a randomly-selected respondent. Never mind that the same respondent could enter repeatedly under different names (increasing their chance of getting the laptop!) the end results of the survey correlated not with anything useful, but with whether the respondent was employed. They were trying to learn something about soft drink preferences but managed to learn nothing at all about that.

Overcoming sampling bias is a hard problem. And, because it’s a hard problem, it’s one of the nails being driven into the coffin of psychology right now: over-dependence on a captive audience consisting of college undergrads. No sampling bias there, for sure!

I can’t find a link to the actual survey so I can’t dig into the survey design itself. That can also be tremendously fun: many survey questions also create or amplify bias: “Do you support the evil gun-grabbing government?” is going to get different responses (I hope!) from “Do you favor gun control: ” a) none  b) limited  c) complete

Ubisoft’s survey results might yield something like: “X percent of users in an online survey that declared themselves to be male identified political correctness in gaming as an influencer in purchasing decisions.”  We’ve all seen how easy it is to pharyngulate an online poll. Who’s gonna take an online poll seriously?

Survey design is hard. But the hardest survey is one where you couldn’t be arsed to test it before you publish it, like Ubisoft apparently did.