Last week, two media polls announced results that seemed almost willful efforts to portray the public as extremists.
The Economist/YouGov poll (8/18/22) announced that “Most Americans see the US/Mexico border situation as a crisis,” with 59% of Americans accepting that characterization, with just 22% who do not. The same day, NPR/Ipsos (8/18/22) reported that “a majority of Americans see an ‘invasion’ at the southern border.”
Both of these polls reinforce the larger narrative being promoted by Fox News and Republicans that the increased number of migrants being stopped at the border represents a serious threat to the United States. And Biden and the Democrats are to blame. Yet the poll results are based on faulty polling questions that seem designed to produce extreme results.
Unbalanced questions
Economist (8/18/22)
The question posed in the Economist/YouGov poll was simple: “Do you think the current situation at the US/Mexico border is a crisis?”
For any experienced pollster, that wording screams of malfeasance. First of all, it is “unbalanced” because it provides no negative counter.
In his 1951 book, The Art of Asking Questions (Princeton University Press), Stanley Payne showed how much an unbalanced question can distort results. In one poll, half the sample of respondents were asked whether in general, manufacturers could avoid laying off workers during slack periods. The other half of respondents were read the same question, but with the added line, “or do you think the layoffs are unavoidable?”
With the unbalanced question, respondents agreed by a margin of almost three-to-one that companies could avoid the layoffs (63% said they could, 22% disagreed). With the balanced question, the margins changed in the opposition direction: 35% said the companies could avoid layoffs, 41% said the layoffs were unavoidable.
That represented a 47-point swing in opinion (41 points in favor to 6 points against), simply by changing the question wording to make it balanced.
And pollsters have known about the problem of unbalanced questions for the past seven decades.
Ensuring an extreme response
The Economist (8/18/22) showed that you can get most people to describe the border situation as a “crisis”—if you don’t offer them any other way to describe it.
But the Economist/YouGov question is also so vague, we really don’t know what people might mean when they agree there’s a crisis. As opposed to what? Do the reporters want to know if people think there is a crisis as opposed to just “serious problems”? If so, they need to provide both options and ask which one the respondents think is more applicable. But they didn’t do that.
They could have asked if people thought there were “serious problems” at the southern border. But how exciting would the resulting headline be? “A majority of Americans think there are serious problems at the US/Mexican border!!”
No. Not worth a screaming headline. Better to ask if there’s a “crisis.” By not providing a counter factual option (such as, “there are problems but no crisis”), the pollsters all but ensured a majority choosing the extreme response.
When I was the managing editor of the Gallup Poll in the early 1990s, we wanted to know how seriously the public viewed the healthcare issue in the country. In 1994, we asked three questions to address that concern, with the results below:
- In your opinion, is there a crisis in healthcare in this country today, or not?
It was an ostensibly balanced question, because it added the “or not” phrase. But it, like the Economist/YouGov question, provided no context. The results showed 84% saying yes to the “crisis.”
- Which of these statements do you agree with more: 1) the country has a healthcare crisis, or 2) the country has healthcare problems but no health care crisis?
Here 53% chose “crisis.”
- Which of these statements do you think best describes the US healthcare system today: 1) the healthcare system is in a state of crisis, 2) it has major problems, 3) it has minor problems, or 4) it does not have any problems?
Just 17% chose “crisis.”
The number of Americans believing the health system was in a crisis went from 84% to 53% to just 17% as respondents were provided with alternative ways of looking at the issue.
The people at the Economist/YouGov poll aren’t necessarily aware of the Gallup results just cited, but they should be aware more generally of the problems with unbalanced and vague questions as outlined decades ago, and certainly included these days in any basic lessons on poll question design.
What is the best way to measure the public’s opinion about the problems at the southern border? The four-part Gallup question about healthcare is a good model for the border issue, because it provides for a fuller understanding of the varieties of views that Americans might hold. Getting at the nuances of public opinion may not provide dramatic headlines, but it’s a more honest way of reporting what Americans are actually thinking.
NPR manipulation
NPR (8/18/22)
The NPR/Ipsos poll is perhaps worse than the one just analyzed. It reeks of manipulation. The question is written in a true/false format, which itself is problematic. And it clearly favors the “true” option.
The poll question: “To what extent, if any, do you believe the following are true? — The US is experiencing an invasion at the southern border.” The answers provided were: “Completely true, Somewhat true, Completely false, Don’t know.”
True/false questions are inherently biased in favor of true, because of a phenomenon called “response acquiescence.” As described in this carefully researched Wikipedia article:
Acquiescence bias, also known as agreement bias, is a category of response bias common to survey research in which respondents have a tendency to select a positive response option or indicate a positive connotation disproportionately more frequently. Respondents do so without considering the content of the question or their ‘true’ preference. Acquiescence is sometimes referred to as “yea-saying” and is the tendency of a respondent to agree with a statement when in doubt….
Acquiescence bias can introduce systematic errors that affect the validity of research by confounding attitudes and behaviors with the general tendency to agree, which can result in misguided inference. Research suggests that the proportion of respondents who carry out this behavior is between 10% and 20%.
In addition, the NPR/Ipsos question is unbalanced, providing two responses for “true” (completely, and somewhat), while only one response for “false” (completely). It’s not at all clear what “somewhat true” means, but if that option is given, a balanced approach would also offer the option of “somewhat false.” But for the “false” option, the question included only the “completely” category.
And then the media pollsters combined the “completely true” percentage (28%) with the “somewhat true” responses (25%) to produce a “majority” of Americans saying there is an “invasion.”
Designed to support a view
NPR (8/18/22) counted people who said it was “somewhat true” to say that there was an “invasion” at the border as agreeing that there was an invasion—although their answer implies that it’s also somewhat false to call it an invasion.
Why did the pollsters ask the question in this biased way? It appears to me as though they wanted to show that “extreme rhetoric” about the border issue has become widespread.
According to the NPR report (italics added):
Republican leaders are increasingly framing the situation as an “invasion.” Immigrant advocates say the word has a long history in white nationalist circles, and warn that such extreme rhetoric could provoke more violence against immigrants.
Still, the polling shows that the word “invasion” has been embraced by a wide range of Americans to describe what’s happening at the border.
Note I said the pollsters wanted to “show” that extreme rhetoric has become widespread, not that they wanted to find out if it was actually the case. They seem to have had their minds made up before the poll was conducted, and designed a questionnaire that would support their view.
Had they wanted to investigate whether most people embraced “invasion” as a way to describe what’s happening at the border, the pollsters could have avoided the simplistic and biased true/false format, and the unbalanced question construction that favored “true,” and instead asked a more objective question.
One such series of question could have been worded this way:
How much would you say you know about what’s happening at the US/Mexico border these days: a great deal, a moderate amount, not much or nothing at all?
From what you’ve read or heard, which do you think better describes what’s happening at the southern border: 1) There is an invasion of immigrants into this country; or 2) There is not an invasion of immigrants into this country, but rather an unusually large number of immigrants seeking legal asylum – or 3) are you unsure?
The first question would allow an analysis to see whether people’s perceived knowledge of what’s happening is correlated with people’s use of “invasion” to describe the border events. The “unsure” option is to let respondents know that it’s okay to admit they don’t know.
In 1994, when Gallup’s polling partners were informed that the percentage of people saying the healthcare system was in “crisis” was only 17% (with the new wording format), the on-air pundit at CNN said he didn’t want to use that question. He wanted a simple result—crisis or no crisis. It was easier, he said, to describe on air. It mattered not that the question forced respondents into extremist positions, even though few people felt that way.
That seems to be the problem with the two polls analyzed here. Dramatic results showing an extremist public are far more “newsworthy” than what the public is really thinking.
The post Manufacturing ‘Crisis’: How Polling on the Border Exaggerates Extreme Opinions appeared first on FAIR.
This content originally appeared on FAIR and was authored by David W. Moore.
David W. Moore | Radio Free (2022-08-25T19:13:16+00:00) Manufacturing ‘Crisis’: How Polling on the Border Exaggerates Extreme Opinions. Retrieved from https://www.radiofree.org/2022/08/25/manufacturing-crisis-how-polling-on-the-border-exaggerates-extreme-opinions/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.