As in 2016, the final results of the presidential election had Donald Trump massively over performing when compared to the expectations based on the results of almost all the major polling outfits. There will be a temptation to beat up on the pollsters who got it so wrong and we can expect Nate Silver and others to be relentless grilled as to why the polls were so off. The ire aimed at them may be less this time because the final outcome of a likely Biden win is looking like it will be correct, unlike in 2016 where the prediction was of a Clinton win. But there will, and should be, much soul-searching as to why the polls were so off again.
Although they were burned by the experience of 2016, before this election many of the pollsters said that it would not happen this time because they had adjusted their samples to include more of the voters who were likely to vote for Trump and had perhaps been undersampled the previous time. Clearly that change did not not make much of a difference.
I think the problem is that Trump has the ability to fire up voters to come out in droves and vote for him and that these voters are also less likely to reveal their intent to pollsters. In other words, the ‘shy Trump’ voter who is enthusiastic about him but is unwilling to reveal that to pollsters is a real phenomenon. As evidence, look at the mid-term elections in 2018 when Trump was not on the ballot. The pollsters predicted pretty accurately the outcomes of the House and Senate races. This time he was again on the ballot and all those who turned up purely because of him also ended up voting for Republicans in down ballot races, thus negating the rosy predictions for Democratic gains in the House and Senate races as well.
While it is true that the Republican party has long been creating a fertile ground for a racist bigot like Trump to take it over, he brings to the role something that I think may not have been anticipated, and that is that a grifter and lying sociopath would be able to create a cult-like devotion among his followers that would allow them to ignore and even gleefully embrace the worst elements of human nature.
Trump is a black swan and statistical models are just not designed to easily incorporate black swans. I am not a statistician but I don’t know if pollsters can take into account the effects of singular phenomena like this. If they had realized after the 2016 polling debacle that he was sui generis, one thing they might have done is make a purely ad hoc adjustment to the polls, by simply taking the difference between the polls and actual results in the 2016 election and modifying this year’s poll results accordingly. That may have got them closer in their predictions to the final outcome. But that would be ugly guesswork and professional statisticians might well be hesitant to do something so crude.
In future posts I will analyze what the results may mean for both Democratic and Republican parties. I think there is room for hope. The Democrats should not over-react to their underperformance in the Senate and House races and start blaming each other as they did today but wait until a more sober analysis is done.