RS: The Breakers Broke: A Look Back At The Fall 2014 Polls
As promised in my first cut after the election, a more detailed walk by the numbers through the 2014 Senate and Governors race polling and my posts on the subject to illustrate that the election unfolded pretty much along the lines I projected on September 15, when I wrote that “[i]f…historical patterns hold in 2014, we would…expect Republicans to win all the races in which they currently lead plus two to four races in which they are currently behind, netting a gain of 8 to 10 Senate seats.”
This was not a consensus position of the models projecting the Senate races at the time; Sam Wang, Ph.D. wrote on September 9 that “the probability that Democrats and Independents will control 50 or more seats is 70%” and described a 9-seat GOP pickup as “a clear outlier event.” On September 16, the Huffington Post model had a 53% chance of the Democrats keeping the Senate, while the Daily Kos model on September 15 had the Democrats with a 54% chance of retaining their Senate majority. Nate Cohn at the New York Times on September 15 gave the GOP just a 53% chance of adding as many as 6 seats, with Republicans having just a 35% chance of winning in Iowa, 18% in Colorado and 18% in North Carolina, and a 56% chance of winning in Alaska. The Washington Post on September 14 had the Democrats favored in Alaska, with a 92% chance of winning Colorado and a 92% chance of winning North Carolina. Even Nate Silver and Harry Enten’s FiveThirtyEight Senate forecast, which was more optimistic than some of the others for Republicans at the time, gave the GOP just a 53% chance of making it to a 6-seat gain as of September 16.
In this case, at least, my reading of history was right, and was a better predictor of the trajectory of the race than the models or the contemporaneous polls they were based on. That won’t always be true; it wasn’t in the 2012 Presidential race. It may or may not prove true in the 2016 Presidential race, where the historical trends overwhelmingly favor Republicans. But after 2012 we were greeted with an onslaught of triumphalism for polls, poll averages and poll models, and what 2014 illustrates is not only that – as we already knew – the models are only as good as the polls, but also that there remains a place for analysis and historical perspective and not just putting blind faith in numbers and mathematical models without examining their assumptions (a point that some of the more cautious analysts, like Silver and Enten, tried to their credit to stress to their readers during the 2014 season).
It also validates my broader view that subjects like polling are best understood when you have an adversarial process of competing arguments rather than deference to a consensus of experts. Because poll analysis down the home stretch involves a high degree of emotional involvement in partisan wins and losses – and most people who get involved in arguments about polling have strong partisan preferences – it’s next to impossible to avoid the pull of confirmation bias, the tendency to credit arguments you want to see win and discredit those you want to see defeated. Certainly mathematical models and poll averages can offer a check against bias, but inevitably they also rest on assumptions that incorporate bias as well. It remains broadly true, as I pointed out repeatedly in and after the 2012 election, that liberal poll analysts and Democratic pollsters tend to do a better job in years when Democrats do well, and that conservative poll analysts and Republican pollsters tend to do a better job in years when Republicans do well, because in each case they are more likely to credit the assumptions that prove accurate. Nate Silver just published a fascinating post on how the 2014 pollsters tended to “herd” towards each other’s results, which tends to exacerbate the problem of being skewed in one or another direction in any given year – more proof of streiff’s view of the herd mentality of pollsters and Erick’s view of polls weighting towards 2012 models without an adequate baseline, and another strike against expert “consensus” thinking and in favor of the virtues of examining your assumptions. The best corrective for the reader to apply to these biases is to listen to both sides, examine the plausibility of their assumptions, and then go back later and evaluate their results.
Continue reading The Breakers Broke: A Look Back At The Fall 2014 Polls