Given the rapidly proliferating number of polls and models predicting the election, the interesting question to be answered after the election is which did the best job. Poll aggregator Sam Wang says that a good prediction should:
Be precise, allowing us to pinpoint a narrow range of outcomes.
Change relatively little in the long term, giving us time to plan in advance.
Give a true sense of the uncertainty – a sense of knowing what we don’t know.
He also looks at the different methods used in the current toolbox of political predictions:
This year, Presidential political predictions have come in multiple flavors:
Purely poll-based (Princeton Election Consortium, Electoral-Vote.com),
Econometric start-of-season models (political scientists),
Mixed (FiveThirtyEight, Votamatic),
Wisdom-of-crowds (InTrade), and
Expert-based evaluations (Charlie Cook, other pundits).
He then gives five criteria for evaluating which one is the best, with the different methods to be scored after the election when the results are known.
Hi,
Last election the 538 blog (as an example) did quite well.
In my recent statistics class the idea was taught that, given regression to the mean, 538 will likely not be as good of a predictor this time.
Is there something I’m missing here?
thanks,
Link is broken.
stevefines: What regression to the mean tells us is that, given 538’s excellent performance in the last round, it is reasonably likely that it won’t do as well this round — but, it is not more likely to do poorly this round than if it had also done poorly last round, in fact just the opposite.
So we shouldn’t be at all surprised if another model does better this time. However, any given model which 538 outperformed last time, we would bet (unless we have strong reason to believe 538’s good performance was a fluke) that 538 would outperform it this time, too — even if we think that one of them will best 538. It’s the Preface Paradox (too lazy to link, but Wikipedia has a good entry): Although somebody has to win the lottery, we are very safe bet against any given person winning the lottery.
OTOH, “regression to the mean” only applies in the absence of other information. I happen to be really impressed by Nate Silver’s methodology, and I expect his model will perform well this time around, too. And for what it’s worth, all of the credible models, as well as the betting markets, seem to be converging on a similar result. Even the aggregate of the econometric models tends to agree.
Off topic, my election prediction was that neither of the leading candidates would mention climate change, Sandy notwithstanding. Bloomberg has dared to speak out but has Obama or Romney commented yet?
I used to look more seriously at the betting markets but I wonder if people who bet money are not basing their bets on these aggregator models and that may explain the convergence. In other words, the betting models are no longer independent guides to the probabilities but are measures of confidence in the models!