To start, there is some grain of truth to the popular conception that non partisan pollsters “won” 2022—I’ve supported that conception before. It is undeniably the case that there were some nonpartisan outlets that were both willing to buck the consensus and saw remarkably accurate results that year. The New York Times and Marist College, for instance, both went against the grain at the end of the cycle and published results that had Senate Democrats ahead by almost the exact margins that they ended up winning by. District-level congressional polling from the Times also proved to be eerily accurate. In my eyes, these two firms gained a ton of credibility here—not just because of how right they were, but because they were demonstrably willing to go against the consensus at risk of being deeply wrong in the same exact way once again. We can, in a sense, trust them to be brave.
However, it’s important to understand that the Times and Marist were both outliers in this regard. For every pollster like them, there were many nonpartisan firms that either didn’t participate in 2022 at all or only published a few surveys. Additionally, there are also quite a lot of nonpartisan pollsters who did participate in 2022, but ended up putting out results that were suspiciously in line with the red wave consensus, even if they weren’t quite as off as the partisan Republican firms. The Washington senate race from that year is a clear example of this. As shown in the linked Times article from 2022, nonpartisan pollsters spent most of the year showing the incumbent Democratic Senator, Patty Murray, up by about as much as she won by. But once Election Day got closer, GOP-aligned firms began polling the race and showing close margins, spurring narratives about the race as a “sleeper flip.” Following this, nonpartisan pollsters who surveyed the race suddenly started to find a far tighter contest. By the end, they, too, would show substantial movement away from the Senator, only finding her up by just over eight points. While it was far more accurate than the nearly-tied race presented by GOP firms, it was still a collective miss of six points. In reality, Murray won by nearly 15.
This phenomenon—wherein nonpartisan pollsters abruptly shifted at the end of the race to match “movement” that only existed in the world of media narratives and GOP-aligned polls —wasn’t just limited to Washington. In New Hampshire, the polls had the incumbent Democratic Senator, Maggie Hassan, up by roughly eight points at the start of October. By Election Day, they only had her up by two. The story was the same as in Washington: GOP-aligned firms started showing a close race near the end, and nonpartisan firms started playing copycat, presenting a race more Democratic than the results from partisan firms but more Republican than the actual results. It also happened in Michigan, just as it did in Arizona. Even in Pennsylvania, which has since been remembered as 2022’s definitive case of GOP zone flooding, much of Oz’s supposed late surge came as a result of pollsters with no connection to the GOP at all suddenly showing leads for him at the end of the race. Noticeably, this occurred just as political talk became dominated by talk of a Fetterman collapse after his debate against Oz.
The most charitable explanation for non-partisan pollsters here is this: that these final shifts represented real movement, but that these states were just tough places to poll, and that they had just been underestimating Democrats throughout. In Washington, for instance, this says that Murray would have “actually” been up by 20 points when nonpartisan pollsters found her up by 13, and that she fell to the margin she actually won by at the close of the election, when they observed her falling from +13 to +8. Quite convenient, but seemingly possible…until you look at the results for this year’s Washington senate primary. In that blanket election, the combined vote of the Democratic Senate candidates beat out the combined vote of the Republican Senate candidates by just under 15 points, a margin practically identical to Murray’s ultimate margin of victory.
This, coupled with the fact that this phenomenon occurred in so many states always right after the beginning of pro-Republican media narratives, should raise some major suspicions at the bare minimum. At the very least, it shows that most nonpartisan pollsters haven’t earned the right to be trusted to be brave like the New York Times and Marists of the world. If they’ve shown us anything, it’s that they’re extremely susceptible to any signs—valid or not—that the GOP could be poised for an upset, and that they see pushing their numbers to the right to be erring on the side of caution. And if they’ve learned anything from 2022, it’s that there are no consequences for doing this. If the GOP ends up outperforming like the partisan polls and the media says they will, they end up being right. But if they don’t, even the biggest Democratic hacks in the world will lay all their blame on partisan “red wave pollsters,” while nonpartisan pollsters writ large will see a reputational boost courtesy of the few firms who were both a) willing to be bold and b) happened to be correct.
This is quite concerning for our specific moment: the end of a presidential election, when the risk of reputational damage is as high as can be and pro-GOP media narratives are in full swing. And it’s extremely concerning when absolutely nothing we’ve seen from the world of pollsters and elite pundits indicates that they’re willing to take any risks whatsoever this year.
…And What we Know is Happening now
For this section, two sources are indispensable: Nate Cohn’s two recent pieces about how pollsters have changed how they operate in 2024. They provide a lot of direct evidence for what was once mostly speculation, and they could very well prove to be a canary in the coal mine if the pollsters end up overestimating Trump this year. But they were not the inspiration for this piece. In fact, what inspired me to publish this article didn’t have anything to do with polls at all. It was when the Cook Political Report changed its rating of Pennsylvania’s Senate race from Leans Democratic to Toss Up, right after they did the same for the Senate race in Wisconsin.
To call these moves cowardly would be a disservice to cowards. They’re also nothing new for the post-2020 punditry world. Of the myriad of examples of their attempts at hedging and ass-covering, the most egregious by far is the one I mentioned at the beginning of this article: when Sabato’s Crystal Ball shifted two entire states towards Republicans so they could project the party as on track to being favored in the Senate while still deferring to John Ralston’s projection that Democrats would hold on in Nevada. It was a complete absurdity—a textbook example of coming to a conclusion first and then working backwards to justify it. And ever since the 2020 election, every forecaster and pundit has done essentially this, working overtime to provide ratings that are the best they can give for the GOP at any given moment.
Understanding this mindset is the only way to understand Cook’s latest moves. Rating those two states as tossups at the presidential level is one thing, but to do it at the Senate level is another thing entirely. These aren’t hypercompetitive races. They’re contests where the Democratic incumbents have held continuous leads ever since they’ve been polled. In Wisconsin, every single poll besides two conducted by GOP-leaning firms has shown a lead for Tammy Baldwin. Things are almost the exact same in Pennsylvania, where only two nonpartisan polls in the entire history of the race have shown anything other than a lead for Bob Casey. Averages currently show Casey up 4.5 points in his race and Baldwin up 3.7 points in hers, both practically identical to the leads currently held by Republican Senators Rick Scott in Florida and Ted Cruz in Texas, respectively.
This might lead one to believe that Cook also has the Texas and Florida races rated as tossups, but they don’t. Cruz’s race is currently ranked as Leans Republican on Cook, while Scott’s is ranked as Likely Republican. It’s a flagrant example of hedging. Unless Cook is willing to be consistent here, in which case their commentary would be so vague as to be useless, there is simply no empirical justification for these ratings other than a belief that the polls are very likely to be very wrong in a way that benefits Republicans. And in this case, you would need to put Trump down as a meaningful favorite to win Wisconsin and Pennsylvania, which Cook doesn’t do. They just want to have their cake and eat it, too, by rating Democrats as low as possible while still providing enough wriggle room to say that they technically never forecasted a Trump win if he loses.
Cook may just be one publication, but their moves provide clear evidence that the elite political class is still well and truly obsessed with one thing: avoiding any overestimation of Democrats at all costs. Here, we don’t even need to speculate that this mindset may be impacting the polls, because we know with pretty strong certainty that it is. Case in point: the two aforementioned recent articles by Nate Cohn for the Times. His first piece, “Two Theories for Why the Polls Failed in 2020, and What It Means for 2024,” goes over the state of discourse among the polling world right now, showing how the industry is still utterly transfixed by its errors four years ago and and is desperate to avoid them once again. At the center of their fear is the supposed problem of nonresponse bias, wherein Trump’s support could be underestimated due to anti-Trump voters of all stripes being more likely to respond to polls than Trump voters. This was something that very much did happen in 2020, and it had a very big impact . But why it happened, and whether it will happen again now, is still far from clear.
Right now, there are two theories purporting to explain what happened four years ago. The first, and the most popular, is that this is simply a problem that’s endemic to polling now, at least whenever Trump is on the ballot. As this story goes, Trump’s unique strength among low-propensity, low-trust voters who pollsters find it very difficult to reach means that they will never be able to properly measure his support. The second and, in my opinion, most plausible explanation is that the fact that this happened in 2020 was mainly because of one big thing specific to 2020: the pandemic. It’s a known fact that lockdowns caused a substantial boost in response rates to pollsters once they began, which would have very obviously resulted in liberals being overrepresented in polling once COVID safety became yet another front on the culture war. This would logically stop being much of an issue once COVID ended, allowing the problem to essentially solve itself.
But even though this debate is hardly settled, it’s clear that pollsters have broadly chosen to err on the side of “caution” by assuming the first theory is true. In response, they have made major changes to the ways they conduct their surveys. According to Cohn in his follow-up article, many pollsters have dramatically changed their process for data collection with an eye on getting the highest possible response rates. Some have attempted to get these high response rates themselves by contacting voters by mail, sometimes providing monetary incentives for respondents. Others have outsourced their weighting to high-profile, high-response “benchmark surveys,” like the Pew National Public Opinion Reference Survey (NPORS). Pew’s most recent survey found a R+2 advantage in public identification, and it has been this number that Cohn says many leading nonpartisan pollsters have used to determine the makeup of their surveys.
It’s an interesting experiment, but one that might have two major problems. The first is that the NPORS was released in July, meaning that it was conducted entirely while Biden was in the race. Knowing how much Biden was individually hurting Democratic chances, it’s entirely plausible that this benchmark number reflects a different political reality than what exists now and is artificially hurting Democratic numbers in the polls. Still, it’s possible that using this data might still be worth the risk if we had clear evidence showing that there’s a substantial risk that traditional data collection methods still have a nonresponse bias benefitting Democrats. This however, gets us to our second problem: some recent data indicates that the opposite may be true. A recent report by the Polarization Research Lab using data from YouGov showed that the proportion of Republicans responding to their survey has gone up as the 2024 election has progressed. YouGov has rightfully responded to this by decreasing their weighting of Republicans in their survey while increasing their weighting of Democrats, which could fix this problem (as an aside, it’s worth noting that YouGov has been one of Kamala’s best pollsters the entire cycle). But if this is something happening industrywide, we have no idea if other pollsters are making the necessary efforts to adjust for it—and the rest of Cohn’s article gives us little reason to believe that might be taking the effort to do so.
Beyond the changes pollsters have made to their data collection processes, Cohn details that pollsters have also made a number of general changes that have the effect of moving numbers in Trump’s direction. The most impactful of these is the decision by many pollsters—two thirds of them, in Cohn’s estimate—to begin weighting polls by “recalled vote.” This is a tough decision to defend on the merits. Its main effect is to just flatly move the numbers in Trump’s direction, a result of the long-observed phenomenon of voters often misremembering who they voted for in the past and just saying that they backed the winner. It’s extremely possible that a sample that self-reports as having voted for Biden by, say, six points could be perfectly representative of the electorate, and that downweighing the sample to match the “real” margin of D+4.5 could just have the effect of giving Trump extra support unnecessarily. It’s not a practice with any track record of success—Cohn noted in an article earlier this month that “weighting on recalled vote would have made the polls less accurate in every election since 2004”—but this may be irrelevant to many nonpartisan pollsters.
Why so? The section at the end of Cohn’s article may give the game away. According to him, one of the major things that pollsters see hope in is the fact that Republican-aligned pollsters make up a higher proportion of polling averages this year than they did in 2020. This alarmed me more than anything else in the article, as there is not a single politically literate person on Earth who looks to Republican-aligned pollsters as a source of accuracy. Those pollsters are complete trash: often headed by election deniers, run as propaganda outlets, and regularly wrong by ridiculous margins. Such firms having a higher presence in the averages isn’t going to do a single thing to make polling more rigorous. All it will do is just move things towards Trump, and this looks to be exactly what many pollsters want.
Self-hating, scared of their own shadows, and liable to see a pro-Trump polling error of practically any size hiding in the bushes, they have made a close election a self-fulfilling prophecy. And even the tools we know they’re employing to shift things rightward might only scratch the surface of what they may be doing. As demonstrated by yet another Nate Cohn study from 2016, the different ways in which pollsters tinker with their results can result in the same exact raw data producing wildly different ultimate outcomes. We can only guess the extent to which they may be using these tools to move things further towards the risk-free, narrative-friendly results that so many of them loved producing in 2022.
Of course, maybe this all just works out. Maybe what we’re seeing about Republican response rates increasing is just a fluke specific to YouGov’s surveys, that Pew’s survey from before Kamala’s entry still accurately reflects partisan leanings, and that all of this is helping us stop nonresponse bias. Maybe there are also further, as-of-now-unidentified causes of pro-Democratic survey error lurking out there, that the extra tools that pollsters are now employing are also helping them stop that. But we shouldn’t take this for granted, because there’s a real history of pollsters becoming too obsessed with their industry’s past blindspots, overcorrecting, and missing badly in the opposite direction. As noted by Nate Silver in a recent article, this happened quite noticeably in the 2017 U.K. election, when a country with a political culture long dominated by the idea of a “shy Tory vote” was just reeling from a 2015 election that saw the Conservatives underestimated in the polls. According to Silver, many pollsters in 2017 put their fingers on the scales to benefit the Tories, often using ad hoc methods to do so. It didn’t end up working out and only caused them to miss very real Labour strength. As a result, all major forecasters but one (the sole exception being YouGov, funnily enough) incorrectly projected a Conservative majority. Pollsters in the U.S. now are facing similar circumstances, harbor very similar self doubts, and are employing equally dubious methods to move their polls in the same direction. They could end up doing something quite similar in the end.
This leads me to my final question: what kind of raw data would pollsters need to start seeing in order to produce polls with a meaningful Kamala lead? They’re clearly very comfortable with producing polls that show her up narrowly with Trump within the margin of error, but what kind of responses would they need to show her up by more than that? They’re clearly capable of imagining a supposedly endemic nonresponse bias that could inaccurately boost her by any amount imaginable. Because of this, it’s entirely possible that they could be converting any dataset they’re presented with to a result within the “safe” band between R+3 and D+3. If Kamala ends up outperforming her polls in the end, we may very well look back on her lack of a surge after events that have historically corresponded with gains, like the DNC and her successful debate, to have been a sign that pollsters were erring on the side of cowardice. This election’s lack of practically any polling variance—something that stands in stark contrast to Trump’s prior elections, including his re-election campaign when Americans had supposedly made up their minds about them—will also stick out like a sore thumb, especially given that one of the major party candidates entered the race at a historically late date while lacking much of a profile to voters. One would think that this would result in a race with a lot of movement, but we’ve hardly seen any since August, around when Kamala started putting together leads close to what pollsters might consider to be safe no matter the result.
In this context, and in light of how many nonpartisan pollsters played me-too with GOP narratives at the close of the 2022 elections, who can we trust to be brave? There are the New York Times and Marist, but they are hardly infallible. While the Times’ eerily accurate closing Senate and House polls massively boosted their reputation in the aftermath of the 2022 election, it’s often forgotten that their final generic ballot poll overestimated Republicans by a few points, or that they editorialized against their own polls that went the furthest against the grain. Sticking by such results when they concern congressional district elections in Kansas is one thing, but it’s another thing entirely when it comes to things like the final poll of a presidential election with Trump on the ballot. Similarly, Marist was hardly free from error in 2022—they underestimated Colorado Senator Michael Bennet’s winning margin by eight points, for instance. They certainly have a degree of credibility that the Emersons of the world don’t have, but they’re not Gods. Even if they were, it’s just never going to be possible to model an entire election off of two pollsters, both of whom are subject to the same incentive structures that all the others are. Pay more attention to them if you please, but don’t expect them to give you a window into the “real” world that other nonpartisan pollsters aren’t showing you.
In any case, we’re well past the point where these decisions won’t have any impact. If they do end up working out and polling happens to be right, the industry will be changed forever, for better or for worse. But if they don’t end up working out, don’t expect it to come without any real-world consequences. We know with certainty now that Trump will declare himself the winner of the election no matter how the results go, and that he and his followers will seize on any bit of proof to claim that it was stolen. In 2020, they were fully willing to use trivia about bellwether counties and the predictive power of Ohio to back up their claim that Trump won. This time, they will be guaranteed to have an extensive list of pollsters showing Trump winning, very possibly for unjustifiably cowardly reasons. In their attempt to cover their own asses, these pollsters may end up giving ammunition to an even more dangerous and well-prepared election denial movement.
We don’t know what the consequences of this may be, but we do know one thing: that, if Kamala wins, Democrats will be too relieved to make fun of the pollsters who messed up. For some surveyors out there, that seems to be all that matters.
spoiler
(Cont’d)
spoiler
(Cont’d)
spoiler