Let me first get this out of the way: I think there’s too much political polling. If you want to forecast the election, you get most of the way using the economic and political “fundamentals.” Polls, when analyzed carefully, provide some additional information, but not enough to justify the saturation coverage they receive in the news media and on social media. Even the most dramatic polls, when interpreted in a careful Bayesian way, don’t tell us much.
Despite that, I think I understand why there’s so much reporting of polls on news and social media. Survey organizations make money doing polls for commercial sponsors, and if you’re doing a national or state poll anyway, you might as well throw in some political questions at the beginning so you can get some press for your org. Also, when the election is coming up, a lot of avid newsreaders want the latest information, so news media will commission their own polls to get some traffic. It all makes sense. But, in a world with a zillion polls, you don’t get much more from the zillion-and-first poll; indeed you don’t get much from the next zillion polls; indeed you don’t get much from the first zillion polls.
From our 1993 paper, Why are American Presidential election campaign polls so variable when votes are so predictable? (incidentally, polls aren’t so variable anymore, but that’s another story), here’s a quick summary:
Our claim is not that fundamentals-based forecasts will always be within 0.3% of the national vote—for one thing, there are many different fundamentals-based forecasts out there; for another, they were off by a couple percentage points in 2000—but rather that, as I said above, they get you most of the way there. I think that a world in which the news media focused on fundamentals-based forecasts and then reported on the occasional poll (recognizing the general level of nonsampling error) would be better than the pre-election world we have now. It wouldn’t change who wins the election; it would just provide a saner basis for reporting during the campaign period.
Unfortunately, in 2008 the polls happened to be pretty much right on the nose, and, this gave poll aggregation a new status, thanks in large part to the work of Nate Silver, first in the primaries and then in the general election campaign. Once the election was over, it turned out that Nate, and other poll aggregators, had forecast something like 49 out of 50 states correctly. Now, most of these states are freebies—you don’t deserve huge credit for predicting that California will go for the Democrats—still, it was a strong performance, just as good if not better than various news media pundits whose job was to handicap the elections.
In retrospect, what made the poll aggregations work was not so much the aggregation as the polls. Polling errors were low in 2008. In 2012, the polls overstated Republican support, but not enough to change the predicted election outcome. Then in 2016, 2020, and 2024 (but not in the off-year elections) the polls overestimated support for the Democrats.
The point is that the success in 2008 led to poll aggregation being a regular part of campaign reporting, and rightly so. There were poll aggregates before 2008, but 2008 was their coming-out party. And, despite the problems with polling, aggregation isn’t going away; there’s just more pressure to improve our aggregation models and appropriately account for uncertainty.
In the meantime, the illusory precision of naive poll aggregation has led many consumers of polls to expect and demand a level of forecasting accuracy that just is in general not possible. The problem is that many of the elections since 2000 have been very close. If polls can be off by 3 percentage points—and they can—then you won’t have a clear sense ahead of time of the vote shares of the two major-party candidates, and this propagates to a big uncertainty in the electoral college. In 2024, there were seven swing states, and various public forecasters (Fivethirtyeight, Nate Silver, the Economist) all gave both candidates a good chance of sweeping all seven and winning more than 300 electoral votes. In an election that could be close, you can’t accurately predict the winner and you can’t even accurately predict the electoral vote total. It’s kind of like a football game with approximately even odds: you don’t have a clear sense of who will win, and your predicted score differential could easily be off by 14 points (see section 1.6 of BDA).
This has led to annoyance at polls and their aggregators, first because the forecasts have been so close to 50-50, and then because the point forecasts were off. As a forecaster, I can very reasonably respond by pointing out that we and others presented wide uncertainty ranges; still, I think we’d be better off without so many polls. Indeed, my main argument for formal poll aggregation of the sort that we and others do is that the polls are out there, people are gonna aggregate them somehow, so it seems reasonable to put a small amount of effort into aggregating them in a sensible way. And when you have a bunch of numbers from different sources, one sensible way is to use statistics, to model the underlying process (in this case, changes in public opinion at the state and national-level) and the measurement process (in the polling context, this is sampling and nonsampling error). So, yeah, I think it’s a good idea to do our best when aggregating information to make forecasts—and credit to those who used more information than we did and did better this time), but, yeah, all these polls are waaaay overhyped.
The polls are what they are. The challenge is to recognize the key strength of the polls (they give a snapshot of public opinion that’s pretty accurate: only off by about 3 percentage points, which is pretty damn impressive given the difficulty of reaching potential voters and getting survey responses), while at the same time understanding their weaknesses (this isn’t enough accuracy to call the winner of a close election, but numbers can give people a false sense of certainty, even when those numbers are accompanied by explicit uncertainty statements.
Lashing out
One thing we’ve seen in 2024, which we saw in 2020 and also in 2016, is a lashing out at the polls and at the quantitative pundits. Some of this comes onto Nate Silver, who’s the most visible representative of polling and forecasting in the news and social media. During the campaign, many liberals were angry at Nate for saying that Biden was getting destroyed and then saying that Harris only had an even chance at winning, despite all the weaknesses the liberals saw in Trump’s campaign. Meanwhile, many conservatives felt that Nate’s 50-50 forecast was a bogus capitulation to a media narrative. Once the election happened, conservatives were no longer angry, but many liberals felt betrayed by the polls and the aggregators who’d forecast a close election and then the Republicans won decisively. To this I respond that (a) the election was actually close—the popular-vote margin was less than 2 percentage points, and a swing of 2 percentage points also would’ve swung the electoral college, which is a close election—, and (b) all the leading forecasters said that a Republican sweep of the swing states was one of the plausible outcomes; the forecasts clearly stated that the election could go either way, and there was never any claim of certainty or near-certainty of a close election (see football score differential example above).
That all said, I can see that, with the election over, conservatives can trash the polls for having a Democratic bias and they can suggest the bias was on purpose (which I doubt, given the commercial motivations of pollsters to be accurate), and liberals can trash the polls for leading them on. My response to all of this is, again, to say that historically polls suffer from some nonsampling bias, which is well understood, and leading forecasters took that into account, and if you ignored the explicit warnings coming from Nate Silver, Elliott Morris (of Fivethirtyeight.com), and others that a 50% forecast probability of winning did not imply an extremely narrow electoral-vote margin, that’s on you.
I do think there are too many polls, but I don’t think it’s the polls’ fault or credit that the election went the way it is, and I appreciate that polls are pretty accurate, despite all their potential sources of nonsampling error.
It’s kinda funny, because when conservatives slam the polls for being biased on purpose in favor of Democrats, they’re implicitly paying a compliment to survey research, as the implication is that pollsters could be more accurate if they just stopped cheating. I don’t think that’s the case. I think pollsters are trying their best to be accurate; it’s just hard to get the level of accuracy that people are expecting or demanding of them. Meanwhile, when liberals label polls as being “useless,” I think they’re being too harsh. A tool is not useless just because it can’t do some unrealistic thing you’re asking of it. Harris was at 51% or 52% of the two-party vote in the polls, and she ended up getting between 49% and 50%—that’s the sort of nonsampling error we see all the time.
Whassup with the gatling gun?
The above is all background. In the title of the post, I promised a gatling gun. Where is it?
The gatling gun comes from Dan Davies, a savvy econblogger who wrote an excellent book on business fraud, among other things.
Davies pointed to a blog post, describing it as “an absolute gatling gun of truth bullets for the political data industry.” The post, by Ben Recht and Leif Weatherby, stated:
2024 was the year the election forecasters gave up. On the Monday before the election, the New York Times polling average showed Donald Trump and Kamala Harris within one point of each other in 6 critical swing states. They put the final election popular vote prediction at 49-48 in favor of Harris. Effectively, a tie. Poll aggregator Real Clear Politics split the difference even finer, predicting the result 48.5-48.5. Poll forecaster Nate Silver put the probability of either candidate winning at exactly 50-50.
OK, let’s unpack this. As I explained last month:
Yes, the forecast was highly uncertain. No, that’s not “giving up.” Let me introduce you to some sports bookies in Vegas. They give lines on every game. Sometimes the two teams are, to the best of all information, evenly matched, and the betting line will be even. That doesn’t mean the bookies “gave up”; it means that, their best estimate is that the two teams are equally likely to win. (OK, not quite, it’s also related to their best estimate of what it would take for equal amount of bets to come in each direction. The point is, they’re not “giving up”; they’re doing their best.)
So, yeah, saying that forecasters “gave up” is just wrong, for the same reason that it’s wrong to label the National Weather Service as “giving up” on days where they announce a 50% chance of rain.
Recht and Weatherby continue:
Whatever happened, it was supposed to be razor thin.
Ummmm, no. Here’s Nate Silver, the best-known forecaster, a couple weeks before the election:
I [Nate] have a guest essay up at the New York Times with a fun headline: “Here’s What My Gut Says About the Election. But Don’t Trust Anyone’s Gut, Even Mine.” . . . Most of the column is about how Kamala Harris could beat her polls — or Donald Trump could beat his again. One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error.
I’m not saying Nate’s always right. We’ve had our disagreements. I’m just saying that, no, he was not saying the election “was supposed to be razor thin.” The forecast electoral vote outcome was a distribution with expected value 50-50 but with a substantial variance.
And here’s Elliott Morris, who ran Fivethirtyeight.com, the best-known forecasting site:
Trump and Harris are both a normal polling error away from a blowout
The race is uncertain, but that doesn’t mean the outcome will be close.
As of Oct. 30 at 11:30 a.m. Eastern, the margin between Vice President Kamala Harris and Trump in 538’s polling averages is smaller than 4 points in seven states: the familiar septet of Arizona, Georgia, Michigan, Nevada, North Carolina, Pennsylvania and Wisconsin. That means that, if the polling error from 2020 repeats itself, Trump would win all seven swing states and 312 Electoral College votes. . . . In a scenario where the polls overestimate Trump’s margin by 4 points in every state, Harris would win all seven swing states and 319 electoral votes. . . .
Both of these outcomes — and everything in between — are very much on the table next week. . . . the model is expecting a roughly 2020-sized polling error — although not necessarily in the same direction as 2020. (In 50 percent of the model’s simulations, Trump beats his polls, and 50 percent of the time, Harris does.)
This point is worth dwelling on. Because our average expectation is for there to be a decently large polling error at least half of the time, there is actually a very low probability that the polls are perfect and the election plays out exactly how the polls suggest. . . . Polls are inherently uncertain. This is why we model. . . . this is the big, fundamental problem with preelection polling: We don’t know what the demographic and political composition of the actual electorate will be, so pollsters are just making the best guesses they can. Those guesses have always, and will always, come with error attached to them.
Here’s some dude who worked on the Economist election forecast:
Why forecast an election that’s too close to call? . . . I think the main value of forecasts is not in the predictions themselves, but in how they portray uncertainty and the stability of the race over time. . . . In the end, elections will always be uncertain, because it is up to the individual to decide how to vote, and whether to vote at all.
And here’s Harry Crane, a forecaster who did better than the name brands in 2024 by incorporating additional information on party registration and early voting:
The forecast makes Trump about a 2-to-1 favorite . . . based on an analysis of fundamental data, polls, and early voting data. This is more or less in line with other opinions out there, such as the betting markets and other forecasters. But because this forecast likes Trump a bit more than markets and a bit more than the other forecasters (who are favorable to Harris), it is inevitable that I will be called an idiot, or worse, should Harris pull it out. The same people will still call me an idiot if Trump wins, so what’s the difference.
The point is that every serious forecaster in 2024 understood—and vocally communicated to the world—that their forecasts were uncertain. Nobody thought Harris or Trump had much of a chance of getting 400 electoral votes, but everybody was saying that 300+ was a possibility. Even the forecasters who were loudly disagreeing with each other on social media agreed on this point, that the forecasts had a lot of uncertainty in the electoral vote.
I find this sooooo frustrating—and I suspect that Silver, Morris, and Crane do too. This is probabilistic forecasting! The point estimate is the center of the forecast; it may be the single most probable value, but lots of other things are possible.
Recht and Weatherby continue:
The result was that, in a “close” race, he won every swing state. That stark truth seems like precisely the sort of thing the prognosticators should have been able to tell us, at least in the aggregate. Instead, the Republicans defeated the pollsters.
OK, 2 things. First, you can put “close” in scare quotes if you want, but the election really was close! The popular-vote margin was less than 2 percentage points, and a swing of 2 percentage points also would’ve swung the electoral college. That’s a close election.
Second, “that stark truth” that Trump (or, for that matter, Harris) could’ve won every swing state was explicitly stated by the forecasters.
Again, soooooo annoying. The “prognosticators” were explicitly telling you that either candidate could sweep all the swing states. You just weren’t listening! You’re criticizing the prognosticators for not telling you something that they were actually saying.
Unless, that is, you wanted the prognosticators to not just say they were uncertain, but to predict the actual election ahead of time. But . . . remember the history of the past hundred years! Polls have nonsampling error. They had nonsampling error back in the days of face-to-face quota sampling, they had nonsampling error in the era of random digit dialing, and they have nonsampling error in the modern era of internet panels. Of course when the polls are so close to 50-50, you can’t use them to make a strong prediction about what will happen in the swing states.
The critics continue:
Low response rates and statistical corrections make polling into a special kind of obfuscated punditry, undermining its claim to neutral objectivity and rendering it useless.
Well, no. As I discussed in my article, Failure and success in political polling and election forecasting, yeah, we’d prefer if polls had no nonsampling error—but nonsampling error of 3 percentage points is not so bad. It just happens that when the forecast is very close, the potential nonsampling error is consequential.
There’s a big difference between imperfect and “useless.”
They conclude that “the survey industry needs a tear-down.” I can’t argue one way or another on that claim. It’s a matter of opinion.
In the comments section, Recht adds, “If you can’t randomly sample, you shouldn’t survey.”
Again, that’s a statement of opinion, so nothing to argue about. There are essentially no random samples of humans, whether you’re talking about political surveys, marketing surveys, public health surveys, or anything else. So if you want to say “If you can’t randomly sample, you shouldn’t survey,” you should pretty much rule out every survey that’s ever been done.
I think that organizations will keep doing surveys, and I think that applied survey researchers will continue to recognize problems of measurement, nonresponse, coverage, and generalization, and they will use statistical models to estimate uncertainty.
I get it that many people are frustrated when news reports focus on point estimates. But in 2024 I think the news media were pretty good about recognizing uncertainty! Even on election day, when forecasts were at 50-50, there were lots of news reports accurately stating that anything could happen. It wasn’t like 1992, 1996, or 2008, when on election day there was a clear expectation of who would win.
So, yeah, it’s too bad that polling isn’t random sampling—it never was!—and a 3 percentage point error isn’t nothing. I just don’t think it’s appropriate to say “the election forecasters gave up” or that “Whatever happened, it was supposed to be razor thin,” given that the forecasters did not say that; indeed they took pains to emphasize their uncertainty.
The real mystery
What I don’t get about the above-linked post is why Davies, who has written so perceptively about problems in the financial system, could take that post so seriously. It’s not a “gatling gun of truth for the political data industry”! It’s actually a mix of falsehoods and opinions.
The big picture
Survey organizations get a lot of press from election polls. Nate Silver and the team at Fivethirtyeight have made a lot of money, and some of that attention has surely come from people who’ve overestimated what polls can say, people who overrated the experience of 2008 when the polls happened to have zero average nonsampling error, people who didn’t read the political science literature so they didn’t know that fundamentals-based forecasts can be more accurate than pre-election polls. Even in 2024, lots of people were taking polls too seriously, which is why I posted that Bayesian analysis of the notorious survey from Iowa.
Given that pollsters and poll aggregators got all this money and fame from people taking polls too seriously, it’s fair enough that will be a backlash, what Davies calls a “gatling gun.”
So, yeah, criticize pollsters and poll aggregators for not delivering what people were hoping for, criticize them for hyping polls in 2008 and coasting on that hype since then. But, for chrissake, don’t criticize them for things they never said. Don’t say, “whatever happened, it was supposed to be razor thin,” given that the leading forecasters took pains to explain that, no, it was very possible that either candidate could sweep all the swing states. Don’t shoot the messenger for something the messenger never said.
Maybe next time use a light saber, not a gatling gun.
P.S. Much of this post appeared earlier but not with the gatling gun connection. It’s just frustrating to me when people take such extreme positions, and then when other people who should know better applaud them for it–which just creates incentives for further hot takes. Just the blogosphere, I guess!
