Islamists Want to Destroy Pyramids »« Oh No! Pat’s Gonna Out Me!

Silver Rates the Polling Companies

Now that his coronation as the King of Polls is finished (though honestly, several other poll aggregation analysts did just as well), Nate Silver compares all of the polling companies to the outcome of the election to see how accurate they were. And maybe the biggest name in polling, Gallup, came out looking pretty bad.

Several polling firms got notably poor results, on the other hand. For the second consecutive election — the same was true in 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark. Mason-Dixon might be given a pass since it has a decent track record over the longer term, while American Research Group has long been unreliable…

It was one of the best-known polling firms, however, that had among the worst results. In late October, Gallup consistently showed Mr. Romney ahead by about six percentage points among likely voters, far different from the average of other surveys. Gallup’s final poll of the election, which had Mr. Romney up by one point, was slightly better, but still identified the wrong winner in the election. Gallup has now had three poor elections in a row. In 2008, their polls overestimated Mr. Obama’s performance, while in 2010, they overestimated how well Republicans would do in the race for the United States House.

He also reiterates that polls that don’t include cell phones underestimate Democratic votes:

The difference between the performance of live telephone polls and the automated polls may partly reflect the fact that many of the live telephone polls call cellphones along with landlines, while few of the automated surveys do. (Legal restrictions prohibit automated calls to cellphones under many circumstances.)

Research by polling firms and academic groups suggests that polls that fail to call cellphones may underestimate the performance of Democratic candidates.

The roughly one-third of Americans who rely exclusively on cellphones tend to be younger, more urban, worse off financially and more likely to be black or Hispanic than the broader group of voters, all characteristics that correlate with Democratic voting. Weighting polling results by demographic characteristics may make the sample more representative, but there is increasing evidence that these weighting techniques will not remove all the bias that is introduced by missing so many voters.

None of this is static, of course. Reputable polling companies are always trying to adjust their sampling models and weighting assumptions to be as accurate as possible. I would expect that in 2014 and 2016, we’ll see more companies doing away with their automated polls and going to live calls (those who can afford it, at least).

Comments

  1. says

    If any polling company were to call my cell phone, the only thing they will get is a piece of my mind: I don’t give a rat’s tail that the people on the other end had no control over what their employer’s machine auto-dialed. I have to pay for those minutes, which makes such calls theft, pure and simple.

  2. Nemo says

    Ah! Finally I understand why I’ve gotten live surveys by people who were such robotic script-readers that they made me think, “They could’ve had a machine do this.”

  3. Matrim says

    @ #1> Uh…not really. If you choose to answer the random number you’re consenting to giving up your minute. If you don’t want to waste minutes, don’t pick up unknown numbers.

    And, while it’s none of my business how you choose to spend or not spend your minutes, unless you’re on some extremely tight pay-by-the-minute plan, the loss of a single minute (which is all you’d lose if you just hung up immediately) doesn’t seem like a big deal. And taking it out on the person on the other end of the line who doesn’t know who the heck you are, what the heck phone you’re on, or which cell phone plan you may or may not have is really tasteless.

  4. abb3w says

    The two obviously significant possible sources of error for Gallup would be errors in their probability weighing for “likely” vs “unlikely” voters (possibly impacted by differences in Democratic and Republican turnout efforts changing the probabilities from historical norms), and the level of weighing assigned to cell phones vs. land lines.

    Oddly, the single most reliable poll this time was the one done based on Snail Mail. Students of polling history may notice some irony….

  5. says

    I got poll-called this year and gave the most surrealistic answers I could. At least I got the poor poll-taker to laugh a couple times. She sounded nice. She was probably one of our secret democratic operatives that we seeded in all the polling companies to skew the results to headfuck the republicans. That op sure worked unexpectedly well! Shazam! Next election we’ll convince them that Bugs Bunny is winning to to watch their heads a’splode.

  6. vmanis1 says

    Marcus Ranum:

    Seriously, why anyone would give a pollster honest answers is completely beyond me.

    I either decline to participate or I answer honestly. The biggest bang I ever got for participating was about 20 years ago. We had a provincial government that was trying to pick a fight with First Nations (aboriginal) tribes over the latters’ still-unresolved land claims. I was called by a polling company, and given a series of questions that were quite obviously loaded to produce a `we support the government’ response; e.g., one of the questions was `what would you do if a First Nations tribe took your house away because they claimed your land’. I answered truthfully, responding in each case with answers favoring a `negotiate fairly’ stance.

    I was one of several hundred or thousand who was called for that poll. Within a couple of weeks, the government withdrew their tough stance, and announced they would negotiate. To be fair, (a) that government soon got slung out because they were stupid, incompetent, and sleazy, and (b) as of 2012, there are still many unresolved land claims issues. But my point stands: by answering political polls honestly, one actually sometimes has an effect on public policy.

  7. abb3w says

    @6, Marcus Ranman:

    Seriously, why anyone would give a pollster honest answers is completely beyond me.

    @7, vmanis1:

    by answering political polls honestly, one actually sometimes has an effect on public policy

    Bingo.

    Honest answers to honest polls let the pollsters’ clients (including politicians) get a more accurate picture of what people’s preferences actually are. This facilitates having a government more accurately representative of the population’s preferences.

    Of course, faced with a poll that says “people prefer X over Y”, there are a lot of ways a politician can react. He can shift his stance to favor X more; he can try harder to persuade more people that Y is (in whatever way) actually the better option than X; he can simply downplay the issue, and try to put X-vs-Y on the back burner so that P-vs-Q gets more notice (and occasionally make slight shifts to shuffle policy more to Y, despite what people want); he can try and field Z as a compromise position, and see how X supporters and Y supporters react to that; and so on. Or, obviously, dismiss the poll.

    Still, polls provide a form of political feedback; like elections, but less decisive.

    Additionally, for atheists in particular, some of the more extended polls gather additional data, that gets aggregated for other studies.

    Though I have to admit, when the Pew Forum asked what my religion was, I was deeply tempted to say “I’m into Mister Magoo/R2D2/Do-as-I-Do Voodoo…”

  8. ottod says

    I will be courteous to anyone who calls me on my landline, as long as they’re equally courteous to me, and to me that means that someone should be listening for me to answer so that they can respond immediately. When a call comes in from a number with blocked ID, I answer it so it won’t roll into my voice mail box. I say, “Hello,” then give a 1 – 2 – 3 count: plenty of time for a real caller to respond. If nobody answers by 3, I hang up. I understand how computer dialing works, but that’s not courteous.

  9. says

    I don’t give a rat’s tail that the people on the other end had no control over what their employer’s machine auto-dialed. I have to pay for those minutes, which makes such calls theft, pure and simple.

    And you then contribute to the mountain of psychological stress inflicted on people that have to take call center work!

    Seriously, I never got this response. It’s analogous to having a bad day due to one person, and responding by punching an unrelated person in the face. Just hang up, no point in ruining another persons day because of their own bad luck.

  10. abb3w says

    @9, ottod:

    I say, “Hello,”

    I prefer the Klingon “nuqneH“, which is a more accurate reflection of my sentiments, and apparently has the added advantage of apparently triggering the “foreign language answer” contingency tree for some computers.

Leave a Reply