Understanding the stability of the polls


One may think that the sensational events of the past week, such as the release of the Donald Trump recording and his ferocious performance in the last debate, would cause major swings in public opinion in support between the two candidates. But in the past, polls seem to shift by only small amounts even with major news breaking and it is not clear that those movements were due to any actual events like debates or are due to random drifts. The latest polls do seem to show more significant shifts with a large drop in support for Trump but if the past is any indication, we may see a regression to the mean and the gap with Hillary Clinton closing again.

Sam Wang continues to remind us that, contrary to expectations with the presence of the mercurial Trump, this is one of the most stable presidential campaigns ever in terms of voter volatility, and he provides charts to prove it. Since Wang started doing this kind of analysis in 2004, the shift in electoral votes projections during the campaign has steadily decreased from 120 in 2004 to 58 so far in this campaign. Wang discusses in this post why he disagrees with other poll aggregators like Nate Silver who argue that there is greater volatility this time around.

wang-electoral-vote-stability

I have been wondering about this and think that the unusual stability may be because of the bizarre candidacy of Trump and not despite it. Such an arrogant, narcissistic, and forceful personality is likely to arouse an immediate response of like or dislike and once that happens is unlikely to change. With Clinton, people have known her on the national scene for a long time and have had time to form a stable opinion that is also unlikely to change.

One can imagine voter sentiment shifting back and forth between two relatively unfamiliar faces (say Tim Kaine and Mike Pence) who are making themselves known to the nation at large for the first time, as people discover things that they like and dislike about each. But Clinton and Trump? Clinton offers few surprises and Trump offers no halfway point. One quickly decides whether one loves him or hates him.

Peter Aldous reminds us of the perils of polling that lead to surprises like the 2015 British elections and this year’s Brexit vote and says that those were likely due, not to sudden shifts in voter sentiment, but to the expense and difficulties of doing random sampling well these days in order to get representative samples. Telephone polling using random digit dialing (RDD) is the gold standard but has become prohibitively expensive and has low response rates and so pollsters resort to quicker and cheaper but less reliable means and this makes polls less reliable. The margins of error that polls quote are based purely on sample size and thus measure random errors and do not take into account systematic errors.

On the other hand, at least for US presidential elections, the sheer number of polls that are being commissioned may provide a counterweight, in that the averages over the polls may wash out some of the large uncertainties of each one.

Comments

  1. Pierce R. Butler says

    … the expense and difficulties of doing random sampling well these days …

    … the sheer number of polls that are being commissioned …

    So – our “news” organizations still spend large amounts of money, but get poorer results?

    Yet another failure of the market system – at best.

  2. sonofrojblake says

    Telephone polling using random digit dialing (RDD) is the gold standard

    Really? What kind of loser actually answers their cellphone to an unrecognised number? Let all the “you have been selected to win £100 in shopping vouchers or an iPad” calls, all the “I’m calling about the traffic accident you’ve had in the last three years that was not your fault”, and all the “we can help you reclaim hundreds in mis-sold payment protection insurance” spam calls go to the answerphone where they can be deleted at leisure. And if random digit dialling calls landlines? Only old people have those things. I’m surprised when/if the polls bear any relation to reality at all, since by definition they’re only polling a self-selecting minority who fall into the category “doesn’t mind answering polling organisation questions”.

    Then again, RealClearPolitics was running a very good averager of polls such as you mention for the primaries, which consistently pointed to a Trump victory pretty much from the day he entered the race… and everyone said it was wrong. Or rather, everyone kept insisting it had to be.
    http://www.realclearpolitics.com/epolls/2016/president/us/2016_republican_presidential_nomination-3823.html

    Encouragingly, currently per RCP polling averages the election is looking good for Clinton, although by no means in the bag.

  3. Reginald Selkirk says

    Really? What kind of loser actually answers their cellphone to an unrecognised number?

    I don’t answer numbers I don’t recognize, but on occasion I do have to answer numbers that are blocked, because my employer blocks numbers on outgoing calls.

  4. kestrel says

    Really? What kind of loser actually answers their cellphone to an unrecognised number?

    The sort of “loser” who is in business pretty much has to. How else are customers supposed to call me? ESP doesn’t seem to work.

    At least I’m not “old”! I don’t have a landline! *score one for me*

  5. anat says

    I do have a landline. My number is unlisted so I hardly get any calls (for a few years I got more calls intended for the couple that had the same number before us than those intended for a member of our household). Though if a pollster manages to call me anyway my likely response would be to hang up the moment I realize what the call is about. I once decided to be a helpful citizen and answer a poll regarding a service I care about and it just went on and on for ever. Not repeating the experience. If the respondent hangs up before the first question is read out, does this contribute to the ‘other, don’t know, refused’ count?

  6. deepak shetty says

    but if the past is any indication, we may see a regression to the mean and the gap with Hillary Clinton closing again.

    I’m hoping Trump loses narrowly and then the Never Trumpers (see if we had a real conservative like Ted Cruz we’d win ) fight with the Trump faction (see if you had supported us we’d have won) leading to a split of the GOP rendering both factions ineffective for a few years hopefully.

  7. John Morales says

    Henry Gale, RDD refers to the method for determining the number to be called, not to whether the dialling is automated or manual. So yes, the method can be used in the US (as indicated by your adduced link), but is more expensive as you note.

  8. Kimpatsu1 says

    “Peter Aldous reminds us of the perils of polling that lead to surprises like the 2015 British elections and this year’s Brexit vote and says that those were likely due, not to sudden shifts in voter sentiment, but to the expense and difficulties of doing random sampling well these days in order to get representative samples.”
    I think there’s another, simpler explanation, Mano: People polled simply don’t tell the truth (with regards to the 2015 UK election, this was called the “shy Tory phenomenon”; i.e., people don’t admit that they intend to vote for a racist, xenophobic party). Another example of this was the Whitewater shopping mall in the Tory heartland of Thurrock, Essex (eastern England). During construction, Thurrock town council polled local residents asking “Rather than clog the roads to the out of town mall with traffic, would you use a courtesy bus service from the town centre to the mall?”
    84% of respondents said yes, and yet the bus sails down Thurrock High Street empty. Meanwhile, people are driving round and round the mall car park desperately trying to find a parking space. Why? Because when people said “Yes, I would use a bus”, what they really meant was “I want everyone ELSE to use a bus so I can drive my car on traffic-free roads”.
    Recently, here in Japan, an absolute majority of unmarried women in their 20s who were polled said that they want to marry and be housewives. Is that really true? Or are they answering the way they know the cranky old misogynists in government WANT them to answer? I know my take on that…

  9. John Morales says

    Kimpatsu1, sure. Polls only work if respondents are honest about their opinion.

    It is true that the OP had an implicit ceteris paribus attached to it, to the effect that the number of dishonest respondents probably haven’t changed over time to a significant degree.

    (One possible factor, but implausible)

  10. Mano Singham says

    Kimpatsu1@#9,

    There are other similar factors at play but they may be less salient. One is the so-called Bradley (or Wilder) effect, where polls over-estimate the support that a black candidate has because people want to show that they are not prejudiced. But that effect has been found to be small to non-existent. In the 2008 election, Obama’s lead in the polls on the day of the election was 7.6% but he won by 7.2%. In the 2012 election, the final polls gave him just a 0.7% lead but he won by 3.9%.

  11. KG says

    The Brexit vote was not a surprise to those who took the polls seriously (as I did). It’s true that most polls in the last few days before the vote indicated a narrow win for Remain, but this had not been the case in the preceding weeks. Overall, results of polling were close enough that a rational observer would have expected a close result – which we got – but have admitted they did not know which side would win.

Leave a Reply

Your email address will not be published. Required fields are marked *