In midterm elections, the president’s party typically loses House seats, especially in the president’s first term, and in particular when the president’s approval rating is low. Given President Joe Biden’s current low approval rating, and that this is his first midterm, the expectation has been for a bad election year for Democrats—as epitomized by the Forbes headline (6/14/22) last summer: “Democrats Midterm Nightmare: Polls Suggest Party Could Face Historic Loss.”
Yet, in the past several weeks, numerous news stories (e.g., Washington Post, 8/27/22; Washington Examiner, 8/20/22; New York Times, 9/16/22) have suggested that the Democrats may not suffer disastrous losses in the House after all.
One of the key indicators mentioned in these articles that portend a possibly more salubrious future for the Democrats is the generic ballot. But reporters have not been especially conscientious in warning readers about how unhelpful that measure can be.
‘The best tool we have’?
The generic ballot is a poll question that asks voters who they will likely support for the House of Representatives in the November election—a Republican or Democrat. It’s called “generic” because no specific candidates’ names are mentioned, only the major party that voters will choose.
538 (6/5/17) considers the generic ballot to be “the best tool we have for understanding how the midterms are shaping up,” and Pew (10/1/02) notes that “it has proven to be an accurate predictor of the partisan distribution of the national vote.” (Spoiler alert: Yes, but not necessarily an accurate predictor of the distribution of House seats, which is what we really care about.)
“Why Was The National Polling Environment So Off in 2020?” asks 538, whose formula rarely works
On September 23, Democrats led in the 538 generic ballot by 1.9 percentage points, though the lead has gone back to 1.3 points as of this writing. RealClearPolitics had Democrats up by 1.3 points recently, but that lead has flipped, with Republicans currently leading by 0.9 points.
How predictive are the generic ballot averages of the actual outcome? History suggests: Not very.
Generic record
There are two parts to analyzing the usefulness of the generic ballot for projecting the political outcome: How accurate is it in predicting the national vote? And how well does the national vote predict House seats?
With respect to the first question, we can look at the records of the two sites that regularly average poll results on this question: 538 (since 1996) and RCP (since 2002). For purposes of comparison, only the data from 2002 onward will be analyzed. Excluding the earlier election years does not disadvantage 538, because its record in those years is a tad worse than its record since then.
Neither site conducts its own polls. They both tabulate averages of polls conducted by other organizations. However, they do average the results in somewhat different ways, so that their final figures, while close, often differ.
Shown below is 538’s own record (2/23/21) of its generic ballot results compared with the actual national vote. (Some of the “national vote” numbers in this table differ slightly from those in 538’s record, but the current numbers are all verified on Wikipedia’s website. For example, here are the 2020 results.
Note that in eight of 10 election years, 538’s polling average over-predicted Democratic strength. In 2020, for example, the final average showed Democrats leading by 7.3 percentage points, but they won the national vote by only 3.1 points—an overestimate of 4.2 points.
538 over-predicted Republican strength only once—in 2008. In 2018, the prediction was right on target.
The last row shows that over the ten elections, Democrats led in the generic ballot on average by 3.5 percentage points, but actually won the national vote by only 1.0 points. The net result is the average 2.4 percentage point over-prediction of Democratic strength. (Arithmetic discrepancies are due to rounding.)
RCP distributes its errors a bit more evenly than does 538. As the table below shows, for six years, the final generic ballot averages over-predicted Democratic strength, while for four years they over-predicted Republican strength.
The 2020 results show similar results between RCP and 538: The former had Democrats leading by 6.8 points, but the party actually won the national vote by just 3.1 points, for a net over-estimate of 3.7 points.
On average, RCP showed Democrats ahead by 2.2 points, though they actually won the vote by an average of 1.0 points—for a net average error of 1.1 points.
The RealClear Politics formula has also frequently missed the mark
National vote vs. House seats won
It’s important to note that the national congressional vote, like the presidential popular vote, has no procedural significance. In the actual election, of course, it’s the vote totals in each of the 435 congressional districts that determine how many seats each party wins. And the national vote doesn’t necessarily predict that distribution.
In fact, theoretically, if the 435 districts were politically homogeneous, one party could win every district with 50.1% of the vote, and win 100% of the House seats. In actual practice, however, the national vote and the distribution of House seats show some recurring patterns.
Shown in the table below is a comparison of the national vote results with the actual number of House seats won.
* Data on the distribution of House seats come from Wikipedia compilations for each year. For 2020, go here.
As the table shows, the national vote has over-predicted Democratic House seats every year since 2002, except for 2008.
In 2020, for example, Democrats won the national vote for the House by 3.1 percentage points, but won 222 seats of the 435, for a 2.0 percentage point margin. The national vote over-predicted their success in the House by a margin of 1.0 percentage points.
The previous year showed the national vote to be even better in predicting House seats. Democrats beat Republicans across the country by 8.6 percentage points, and won more House seats than Republicans by an almost identical 8.3 points.
But that close match between the national vote and House seats was not at all expected.
Trigger warning 2018
In fact, in 2018, given the previous three elections, when the national vote over-predicted Democratic strength by 9.7, 7.9 and 8.7 percentage points respectively, some reporters issued a trigger warning about the generic ballot. It wouldn’t be enough for Democrats to win the national vote by one or two points, they would note. Democrats would have to win by seven or eight points just to break even in the number of seats.
Once such caveat came from Nate Silver of 538 (6/5/17), who, in the year before the midterms, argued: “A final generic ballot average showing Democrats up 7 points would suggest that they’d be about a 50/50 proposition to take back the House.”
The nonpartisan Brennan Center for Justice had an even more cautious view, suggesting those figures were likely to be far off the mark. In an extensive report, “Extreme Gerrymandering and the 2018 Midterm,” scholars at the Brennan Center made these observations (italics added):
Maps drawn after the 2010 Tea Party wave to favor Republicans, particularly in big swing states like Michigan, North Carolina and Ohio, mean Democrats would need to win the national popular vote in 2018 by the biggest margin in a midterm since 1982….
To attain a bare majority, Democrats would likely have to win the national popular vote by nearly 11 points.
The actual election results surprised everyone: Democrats won the national vote by 8.6 percentage points, and they won more House seats than Republicans by an almost identical margin of 8.3 percentage points.
What happened to that gerrymandered advantage the GOP was alleged to have had? It appeared as though the Trump presidency was especially effective in energizing Democrats to turn out, overwhelming whatever districting disadvantage they faced.
Again, in 2020, the national vote percentage was close to the percentage of seats won by Democrats: They won the vote by a 3.1 point margin, and had a 2.0 point margin in seats won.
Generic Ballot and House Seats – 2002-2020
Overall, the generic ballot has not been especially accurate in predicting how many House seats each party has won. As shown above, both RCP and 538 have typically over-predicted Democratic results in the national vote. And the national vote almost always over-predicts the number of House seats Democrats win.
When the errors are added together, it shows how far off the generic ballot predictions actually are. Shown below are two tables—one that compares the percentage of House seats won with the 538 generic ballot averages, the other that makes the same comparison with RCP data.
For journalists using the generic ballot to estimate how the parties might fare in the midterms, this table should provide a clear warning. Only once in those ten elections did the 538 generic ballot averages come close to predicting the outcome of the election. That was in 2018. In every other year, the discrepancy between the generic ballot and the outcome was greater than 4 percentage points.
The “absolute error” averaged 7.4 points, with most of the error over-predicting Democrats (by an average of 5.5 points).
The RCP averages were somewhat more accurate, with an “absolute error” of 6.2 percentage points, and a biased error (in favor of Democrats) of 4.2 points. The latter figure was the result of RCP showing Democrats ahead on average by 2.2 points, though they lost on average by 2.0 points.
The predictions in two years—2018 and 2010—were within two points of the actual results. Still, in eight out of 10 elections, RCP’s predictions were off by 3.8 points or greater.
Scatterplots of 538 and RCP Generic Ballots with House Seats
Another way to illustrate the relationship of the generic ballots and House seats is through the use of scatterplots.
The 538 scatterplot below shows a fairly strong correlation between the two measures (R2=.69), but at the same time reveals significant errors.
The two most recent elections—2018 and 2020—are pretty close to the projected line, but other years—such as 2006, 2008, 2010, 2012, 2014 and 2016—are all four or more points above or below the line.
The regression formula suggests that for Democrats to have a 50/50 chance of winning the House, they would need to lead the 538 generic ballot by 5.0 points (where the regression line crosses the horizontal zero line).
The RCP scatterplot is similar, with a slightly higher correlation (R2=.72).
This regression model suggests Democrats would have to lead Republicans by 3.6 percentage points in order to have a 50/50 chance of winning the House.
In each case, the standard error of the estimate is substantial, suggesting the projections could be wrong in either direction by several percentage points.
Predicting the 2022 midterms
The scatterplots provide a good visual illustration of how well the generic ballots predict House seats. These regression models, however, are only slight improvements over the earlier tables that show the average error in prediction. For 538, the tables suggest an average error of 5.5 percentage points, while the regression model suggests the projection will be off, on average, by 5.0 points. For RCP, the comparable figures are 4.2 and 3.6 points.
Nate Silver of 538 (9/16/22) admits his House model “probably does overstate the case for Democrats.” He went on to suggest that the model
corresponds to current polls overstating Democrats’ position by the equivalent of 1.5 or 2 percentage points. Put another way, we should think of a race in which the polling average shows Democrats 2 points ahead as being tied.
But that’s two points ahead in the national vote. Add to that the 3.0 percentage point average discrepancy between the national vote and percentage of House seats, and we’re back to thinking about a race in which the polling average shows Democrats five points ahead as being tied.
And even then, the projection is likely to be off by several points.
Perhaps the best advice for reporters assessing the likely outcome of the midterms is expressed by Walter Shapiro on Roll Call (9/27/22), who suggests that 2022 is an unusual election year that does not fit the pattern of election years past (“You just can’t account for the weirdness of 2022”).
His advice: “For those tempted to predict the midterms—don’t.”
The post The Persistently Faulty Record of Generic Ballot Polling appeared first on FAIR.
This content originally appeared on FAIR and was authored by David W. Moore.
David W. Moore | Radio Free (2022-10-03T16:08:05+00:00) The Persistently Faulty Record of Generic Ballot Polling. Retrieved from https://www.radiofree.org/2022/10/03/the-persistently-faulty-record-of-generic-ballot-polling/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.