POLL POSITION

Bogus Poll Statistics

By Allan Rivlin
NationalJournal.com
Thursday, Nov. 13, 2003

The past few weeks have given us several good examples of survey results becoming part of the collective political wisdom because they support one side's ideological beliefs -- despite failing to meet the standard of being well-reasoned conclusions from statistically valid surveys.

Perhaps you have read that recent polls have found:

-The Fox News Channel misinforms its viewers, causing them to flunk a test that gets aced by people who turn to public TV or radio for their news.

-Morale is so low among the ground forces in Iraq that as many as half of the troops say they do not expect to re-enlist.

-Iraqis want the United States to stay in their country until an American-style democracy is established.

All of these may be true, but none are proper interpretations of rigorous survey research.
Really they are just three new examples that demonstrate the proven academic theory that people tend to evaluate survey results based on whether they agree with the conclusion first, without ever really asking if the data in fact come from a valid survey. People hold onto poll statistics that support their positions regardless of the quality of the underlying research, or in one case, whether the underlying research actually supports the opposite conclusion.

Fair And Biased?
Left-leaners would love to have proof that the Fox News Channel isn't "fair and balanced," but instead just plain inferior. So naturally, they were interested in the findings from a study that showed people who primarily watch Fox News are significantly more likely to have misperceptions than people who watch other networks -- especially PBS and NPR, whose viewers and listeners held the fewest misperceptions.

The finding comes from a new analysis of a series of surveys administered over the Internet throughout 2003 by Knowledge Networks for the Program on International Policy at the University of Maryland. The report and press release are fairly careful in how they describe the findings, but the further you follow the thread on left-leaning blogs, the more unfair the assertions become.

One typical headline: "It's proven: Fox News Makes You Dumb!" It is safe to guess that the person who wrote that was a little skeptical about the Fox News Channel even before reading the PIPA study.

But there are several reasons why the bloggers should be cautious about throwing these findings in Brit Hume's face.

-The fact that this is an Internet poll does not turn out to be the greatest cause for caution. Knowledge Networks builds its panel using random-digit telephone recruiting, and gives respondents the equipment to access the surveys, answering most of the concerns about Internet surveys.

-However, there is no clear way to know how many of the viewers who say they rely on Fox News ever watch the 24-hour cable network as opposed to watching the local news on their Fox affiliate. They just said they rely on Fox News instead of CBS, ABC, NPR, PBS and NBC (MSNBC, local news, the "Today Show"? We don't know.).

-With just 3 percent (fewer than 50 people) saying they rely on PBS or NPR, we can forget reliable proof of our, er, their superiority.
But the biggest problem is that the survey finds the political right was wrong about what the left considers to be the war's most important facts, and conservatives are more likely to watch Fox News. All of the "misperceptions" they track are correlated with political ideology, with support of the president on the "wrong" side.

It is possible that PBS viewers are less well-informed than the FNC crowd when it comes to facts about Saddam Hussein's torture practices. But if you believe Saddam had ties to al-Qaida, that the United States found weapons of mass destruction in Iraq, or that world public opinion backed U.S. action, you must be stupid or watching the wrong nightly news.

A secondary finding from this study about news sources caused it to get more attention than its primary findings about misperceptions and the correlation to support for the war. This is because some people are hungry for rigorous proof that FNC is unfair and unbalanced, but this is not that study.

Stars, Bars And Pie Charts
So you're reading the Washington Post and for the first time you can remember, it is giving you the results of a survey conducted by Stars and Stripes. First you read that "half those questioned describe their unit's morale as low and their training as insufficient, and said they do not plan to reenlist."

It is not until the middle of the third paragraph that you are told the results were not obtained through scientific methods. I am sure you stopped reading the article right away. In truth, if you had stopped you would have missed something of value. A bad survey can nonetheless be very good reporting.

Typically reporters talk to a handful of people to write a story. Stars and Stripes sent three teams of reporters who talked to large numbers of troops and passed out thousands of questionnaires. Nearly 2,000 were returned.

A lede paragraph that was faithful to the facts might have read something like: "Stars and Stripes reporters talked to nearly 1,000 service men and women in Iraq who described their morale as low and their training as inadequate. They also talked to similar numbers who said their training was strong and their morale high..."

Because this is not a scientific poll, the results cannot be projected onto the population, so it is inappropriate to report percentages. It is entirely possible that disgruntled soldiers are more likely to take a questionnaire and return it. Even if we have no basis to say they are in the majority, the sheer numbers of unhappy GIs the reporters met is certainly news that was worth reporting. But...

Cheney's Use Of Zogby Poll Disputed By Zogby
It is one thing to overdraw a survey's conclusions. It is something else to attempt to draw the opposite conclusions than the data suggest. That's what John Zogby says Vice President Dick Cheney has been doing with Zogby's poll of Iraqis.

On NBC's "Meet the Press" on Sept. 14, Cheney said the Zogby International poll had "very positive news in it." He said that given five choices for a government to emulate, "the United States wins hands down." And that "asked how long they want the Americans to stay, over 60 percent say for at least a year."

The pollster had a quite different interpretation of his poll. "It is very difficult to find any good news for the United States in the poll," Zogby said this week, and he started to list the findings Cheney either failed to mention, glossed over or got wrong:

-By a 50 percent to 36 percent margin, Iraqis say the United States will hurt rather than help Iraq over the next five years. Majorities say the Saudis and the United Nations will help.

-By 51 percent to 39 percent, the Iraqis surveyed rejected the statement, "Democracy can work in Iraq," in favor of the statement, "Democracy is a Western way of doing things that will not work here."

-Given a choice of five countries for Iraq to model its government on, 23 percent chose the United States, 17 percent chose Saudi Arabia, followed by Syria (12 percent), Egypt (7 percent) and Iran (3 percent).

Zogby does not agree with Cheney's interpretation of 23 percent saying they think Iraq should emulate the United States. "When the vice president said on 'Meet the Press' 'they chose the United States hands down,' uh-uh." Zogby said. "No, they didn't choose the United States hands down."

Zogby also disputes Cheney's statement that the survey shows more than 60 percent want the United States to stay for at least a year. Here is the poll question and the results:

Given a choice would you like to see the American and British forces leave Iraq in six months (32 percent), one year (34 percent), or two years or more (25 percent). One could interpret these results as a broad mandate to stay one year or more, but it would be a mistaken interpretation.

And it would be hard to square that interpretation with the finding that only 32 percent say America and Britain should help make sure a fair government is set up in Iraq. Sixty percent say they should just let Iraqis work this out for themselves.

What all of these examples have in common is what psychologists call the theory of cognitive dissonance. We are all more welcoming of perceptions that fit our pre-existing views and critical of perceptions that do not. But this final example has consequences that are potentially even tragic.

Remember, President Bush has made it clear that he does not typically read the news but rather relies on advisers to provide him with an "unbiased" account. If this is an example of Cheney's critical reading skills, then America's path into this war becomes a little easier to understand.

Allan Rivlin, a NationalJournal.com contributing editor, is a senior vice president of Peter D. Hart Research Associates, a Democratic polling firm. His e-mail address is arivlin@nationaljournal.com.