How Media Reports of Their Own Polls Can Mislead

Investor’s Business Daily (1/10/22)

A new media poll last week by Investor’s Business Daily (1/10/22), conducted with the polling firm TIPP, announced that “Biden Approval Rating Relapses as Omicron Surges, Stock Market Slumps.”

There are at least two problems with the Murdoch-owned outlet’s announcement.

First is that the “relapse” in approval rating is reported to be “nine-tenths of a point.” While the author, Jed Graham, does not specify a margin of error, we can safely say that a difference of less than 1 percentage point clearly falls into the category of random variation. One cannot reasonably conclude, therefore, that there has been a significant change in opinion.

A second problem is that the standard way of reporting the IBD/TIPP poll results shows Biden’s approval rating actually increasing by 1 percentage point over December. (That difference is still not significant.)

Creating a ‘relapse’

How, then, did the poll report a “relapse”?

The poll report excluded respondents who were unsure or refused to give an opinion, which changed the base. In doing so, the poll reported a 50.1% approval in December, and a 49.2% rating in January—which produced the mentioned decline of 0.9 of a percentage point.

Typically, pollsters do not exclude “unsure” respondents, because to do so is to distort what public opinion actually is. People who are unsure still constitute a significant portion of the public, and should be included in the analysis.

When IBD/TIPP follows this standard approach, its January poll shows 44% of adults who approve of Biden’s job performance, 45% who disapprove, and 11% with no opinion. When the IBD (12/6/21) released its December poll (12/1–4/21), the comparable figures were 43% approve, 43% disapprove, 14% no opinion. Using those figures, one would conclude Biden’s approval increased by one point over December.

The author of the article chose to focus on the decline in approval of (essentially) 1 percentage point, rather than the increase. And he probably did so because the lapse fit in with the story that Biden was being hurt by the spread of Omicron and the declining stock market. It would have been a more awkward story to note that Biden’s approval had actually increased during these troubling times. How to explain that?

Other polls unmentioned

A related problem is that the author fails to mention trends reported by other polls, as though the IBD/TIPP poll is the only poll to measure Biden’s approval rating. If a news organization is going to report on the beat of public opinion, certainly other polls are relevant sources to examine.

But reporting on other polls is not the norm when media organizations sponsor their own polls. Instead, each organization tends to do what Investor’s Business Daily did—treat their own poll as the definitive description of what the public is thinking.

The author of this report does note that while the IBD/TIPP poll found a 44% approval rating (using standard methods of reporting), the RealClearPolitics average of other polls at the time had Biden’s approval rating two points lower (42.1%). The author does not, however, acknowledge that the trend in Biden’s approval rating essentially showed small fluctuations with no significant change—which would have undermined the central theme of the story.

Reports that mislead

USA Today (11/7/21)

Investor’s Business Daily is not alone in failing to account for other poll results. In early November, USA Today (11/7/21) announced the results of its poll (11/3–5/21) showing President Biden’s approval rating at a “new low” of 38%. On the generic ballot—a question that asks whether a voter would choose a Democrat or a Republican in the 2022 congressional election—Republicans outperformed Democrats by an 8-point margin, 46% to 38%.

A week later, ABC News/Washington Post (11/14/21) reported that their poll (11/7–10/21) also found a “new low” for Biden’s approval rating of 41%, and a record-setting Republican advantage on the generic ballot of 10 points, 51% to 41%.

Ten days later, NPR (11/24/21) reported the results of its poll (11/16–19/21) showing Biden’s approval rating with, yes, a “new low” of 42%. Only in the tenth paragraph does NPR acknowledge that its generic ballot measure nevertheless shows Democrats ahead by “a slight 46%-to-41% advantage.”

All three poll reports were seriously misleading. First, there were not three times that month when Biden’s approval rating reached a “new low.” Second, all three polls produced outliers on the generic ballot, misleading their consumers as to a more plausible picture of public opinion.

Aggregating results

None of the three poll stories mentioned the results of other media polls. Each report treated its own media poll as though it alone has the current temperature of the American public. Yet, because of statistical variation, any given poll can be off the true mark. An average of credible polls is one way to correct for variable results from individual polls.

As noted by the American Association for Public Opinion Research, referring to electoral polling:

The benefit of a poll aggregator is that an average of polling results should give a reporter a more reliable and wider perspective on a race, rather than relying on just one poll’s result.

For example, because of variation among the polls, looking at the average across a number of polls to characterize the presidential race…would provide a more complete and more stable picture than looking at the result of just one poll.

There are at least two major poll aggregator websites—538 and, as noted earlier in this article, RealClearPolitics (RCP)—which routinely provide averages of major polls about both Biden’s approval ratings (538 and RCP) and the generic ballot (538 and RCP).

Here I focus on 538, because it adjusts the impact of polls based on the quality of the polling organization, and because it screens polls to insure they meet minimum standards. Still, the differences between the two sites are minor.

Compared to the average

During the three day period that USA Today found Biden’s approval rating at 38%, 538 showed an average 42.9%, essentially five points more positive. The USA Today poll was a clear outlier compared with most other polls. Of course, the newspaper did not mention that fact.

Presidential approval polling averages (538, 1/18/22)

The other two polls, showing Biden’s approval rating at 41%–42%, were in the ballpark (within 2 percentage points) of the averages at the times of the interviews, but it would be incorrect to describe either result as a sudden “new low”—though that’s how the poll results were characterized.

As for the generic ballot, all three polls produced significant outliers. ABC News and the Washington Post reported Republicans with a 10-point advantage. The USA Today 8-point GOP advantage was almost as dramatic.

Averages by 538, however, showed Democrats actually in the lead during the interviewing periods of both polls—by 2.0 points during the USA Today polling period, and 1.1 points during the ABC/WP polling period. The net differences between the averages and the individual polls were 10 and 11 points, respectively, in favor of Republicans.

Generic congressional ballot polling averages (538, 1/18/22)

The NPR poll, showing Democrats leading in the generic ballot by five points, was an outlier in the opposite direction. During its polling period (11/16-19/21), 538 showed an average GOP lead of 0.5 points—a net difference of 5.5 points.

A more realistic view

As the 538 averages suggest, there was hardly any movement in Biden’s approval rating during November, and there was only a modest movement in the generic ballot.

In contrast to the three polls, which suggest a sudden drop in Biden’s approval rating in early November, recovering to about 41%–42%, the 538 averages suggest that Biden’s approval rating remained almost constant during the whole period, varying only within a half of a percentage point. That pattern prevailed into December and early January, when IBD/TIPP conducted its polls.

On the generic ballot, in contrast to the three polls that show Democrats trailing by 8 and 10 points in early November, then dramatically recovering to a five-point lead in mid-month, the 538 averages show the generic ballot moving from a modest 2.7-point advantage for Democrats in early November to less than a 1 point advantage for Republicans in mid-November.

Why not cover other polls?

From a journalistic point of view, there is no excuse for a news organization to report the results of its own poll and ignore similar results already published.

News media justify conducting their own polls as covering the “beat” of public opinion. As ABC News explains (italics added):

At the ABC News Polling Unit, we are news reporters first; we think of public opinion as our beat—like covering the Supreme Court, the White House or the Pentagon. In many ways the process is the same: We pick a topic, formulate questions, go to our best sources, ask what we need to know and report what we’ve learned.

The requirement to go to the “best sources” must certainly include not just the news organization’s own poll results, but those by other reputable organizations as well. And RCP and 538 provide journalists with an easy reference to such other sources—at least for the president’s approval rating and the generic ballot.

Of course, each media poll is likely to include numerous questions not aggregated by RCP and 538. But that’s no excuse for the reporter to exclude contemporary trends when they are available.

Still, it’s understandable why most media poll reports do so. The news organization has just invested a considerable amount of money to conduct its own poll, and it certainly doesn’t want to undermine its results by showing how far off they are from a poll average.

Had IBD/TIPP reported the RCP trend showing essentially no change in Biden’s approval rating, that would have undermined their story that linked the surge in Omicron to Biden’s decline in approval.

It is what they say it is

ABC (11/14/21) announced that “Republican congressional candidates currently hold largest lead in midterm election vote preferences in ABC News/Washington Post polls dating back 40 years.” Not mentioned: Most polls at the time found an advantage for Democrats.

Another example is ABC News (11/14/21), which made much of its findings on the generic ballot:

That’s the biggest lead for Republicans in the 110 ABC/[Washington] Post polls that have asked this question since November 1981. Indeed, it’s only the second time the GOP has held a statistically significant advantage (the other was +7 points in January 2002) and the ninth time it’s held any numerical edge at all.

It would be hard to imagine the report then saying, “Actually, our results are way out of line with other polls, which show an average advantage to Democrats. Our record GOP lead is probably an outlier.”

Admitting that their polls may not be the best measure of public opinion, and in some cases may in fact be seriously flawed, would undermine the credibility, not just of the generic ballot measure or Biden’s approval rating, but of all the other questions in the poll as well.

In short, there is a financial/corporate incentive for media poll reports not to tell the whole truth about the state of public opinion. As far as the news organization is concerned, regardless of what other polls show, public opinion is what their new poll says it is.

This is, of course, a reason not to put too much weight on any single report about an outlets’ polling results.

The post How Media Reports of Their Own Polls Can Mislead appeared first on FAIR.

This content originally appeared on FAIR and was authored by David W. Moore.

Print Share Comment Cite Upload Translate Updates