Big Lizards:Blog:Entry “A Tale of Two Surveys”

« The Greater of Two Lessers | Main | Dixie Chicks - Not Ready to Make Amends »

May 15, 2006

A Tale of Two Surveys

Hatched by Dafydd

Or, Creatively Crafting Crafty Questions For Fun and Profit.

Two recent surveys asked respondents about the National Security Agency and the "recent" revelations -- actually four and a half months stale -- that they have collected data from various phone companies on calling habits, creating a massive database to be used to backtrack terrorist connections and spot al-Qaeda plots before they happen.

In the poll by ABC News and the Washington Post (free registration required), 63% of Americans found the NSA call traffic analysis program "acceptable," while 35% found it "unacceptable;" but in the Newsweek poll, conducted at the same time, 41% called it a "necessary tool," while 53% said it "goes too far." What gives? How can two surveys come to virtually opposite results?

Who Gives the Orders Around Here?

There are a number of factors that affect the responses. One is question order: the Newsweek poll first asked whether the respondents approved of how Bush was handling the war on terrorism, then immediately jumped into the NSA question.

Since "terrorism and homeland security" is inextricably bound up with the war in Iraq, and since anyone who follows the antique media version of that war is convinced we're "losing," the "terrorism and homeland security" question is calculated to elicit a negative reaction. Leading with a question that eliicits a response negative towards the president typically causes all subsequent questions about presidential programs to be similarly negative. It's human nature; the earlier questions put respondents in a foul mood.

By contrast, the ABC/WaPo poll asked first whether respondents believed that Bush was protecting Americans' right to privacy while fighting the war on terrorism (which gets a marginal Yes vote); then it asked whether it's doing enough to protect Americans' rights in general (another Yes, this time somewhat stronger: Americans aren't big on rights when their personal survival is at stake). Then which is more important... investigating terrorist threats or protecting privacy (this one is a no-brainer, and it gets a 65-31 victory for investigating terrorism... building to a crescendo).

And only then does it ask about the NSA program. Since respondents have already been primed to compare safety with privacy, they're certainly going to be inclined towards the former.

But there is a much more direct reason why the ABC/WaPo poll found so much more support for the NSA program than did Newsweek.

Words to the Wise

More often than not, the most determinative factor in a poll's results is the actual wording of the questions themselves. Happily, the link above to the ABC/Washington Post poll includes the questions as well as the answers. I cannot find a direct link to the actual questions of the Newsweek poll; but PollingReport.com has the specifics (for a limited time -- get 'em while they're hot!)

Here are the two polls' questions about the NSA program. See if you can figure out the huge, gaping distinction between them. For extra credit, there is one more subtle point that might also affect the responses.

Newsweek poll:

As you may know, there are reports that the NSA, a government intelligence agency, has been collecting the phone call records of Americans. The agency doesn't actually listen to the calls but logs in nearly every phone number to create a database of calls made within the United States. Which of the following comes CLOSER to your own view of this domestic surveillance program? It is a necessary tool to combat terrorism. It goes too far in invading people's privacy.

ABC News/Washington Post poll:

It's been reported that the National Security Agency has been collecting the phone call records of tens of millions of Americans. It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism? Do you feel that way strongly or somewhat?

To find out whether you were right, slither on to the extended entry!

Here are the two questions again, this time with the differences highlighted:

Newsweek poll:

As you may know, there are reports that the NSA, a government intelligence agency, has been collecting the phone call records of Americans. The agency doesn't actually listen to the calls but logs in nearly every phone number to create a database of calls made within the United States. Which of the following comes CLOSER to your own view of this domestic surveillance program? It is a necessary tool to combat terrorism. It goes too far in invading people's privacy.

Points to note: the most blatant omission is any explanation of the connection of this program to international terrorism! If a respondent wasn't closely following this case, he might have no idea in the world that the purpose of the NSA program is to spot calling anomalies that might indicate a connection to international terrorist organizations like al-Qaeda or its affiliates.

For all he knows, this could be purely "domestic surveillance" by President Bush of prominent congressional Democrats or anti-war activists. Given the choice between domestic surveillance (of political opponents) and privacy rights, of course more Americans are going to choose the latter. Who wants another Nixon?

The more subtle, extra-credit point to note is that the question does not spell out just what the "NSA" is, other than identifying it as "a government intelligence agency." Few people have heard of it; it's not a political celebrity like the CIA.

When you go to work tomorrow, ask your coworkers what the NSA is; and just call it by its initials. I'd love to know what percent of people can give a reasonably good explanation. It is, of course, the National Security Agency, the largest intelligence organization in the United States and with the biggest budget. It engages in "signals intelligence," eavesdropping on electronic communications and operating spy satellites in orbit.)

ABC News/Washington Post poll:

It's been reported that the National Security Agency has been collecting the phone call records of tens of millions of Americans. It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations. Would you consider this an acceptable or unacceptable way for the federal government to investigate terrorism? Do you feel that way strongly or somewhat?

In counterpoint to the "subtle point" above, in this question, the name of the agency is spelled out: respondents learn that it's about "national security," which makes it clear to most (I believe) that it's not a domestic intelligence agency (as the previous question clearly implies).

But more important is that this question actually explains the connection within the program between recording the number, length, and target of the calls and national security: "It then analyzes calling patterns in an effort to identify possible terrorism suspects, without listening to or recording the conversations."

The point here is quite obvious to me: when Americans have no idea why some seemingly private data (it's really not; but that's a whole 'nother argument) is being collected by the government, they're instinctively against it. This is a healthy trend that you likely wouldn't find in Europe, where government intrusion is readily accepted (not a single European country was founded in rebellion against a colonial master).

But when you actually explain the purpose behind the NSA traffic-analysis program, Americans think it's a great idea and support it two to one.

And that, in my opinion, is the most likely explanation for the seeming disparity of the poll results: Newsweek didn't bother explaining why the NSA was doing this... and ABC and the Washington Post did.

UPDATE 03:18: Well, that was fast: no sooner did I publish this post than I saw that tomorrow's USA Today/Gallup had a similar survey that found "a majority of Americans disapprove of a massive Pentagon database containing the records of billions of phone calls made by ordinary citizens."

The "majority" in question is 51% (to 43% approving). Here is the question they asked:

As you may know, as part of its efforts to investigate terrorism, a federal government agency obtained records from three of the largest U.S. telephone companies in order to create a database of billions of telephone numbers dialed by Americans. How closely have you been following the news about this?

Based on what you have heard or read about this program to collect phone records, would you say you approve or disapprove of this government program?

The USA Today/Gallup survey does not explain the connection... and it gets results eerily similar to the Newsweek poll, which also fails to explain the connection. 'Nuff said.

Hatched by Dafydd on this day, May 15, 2006, at the time of 3:09 AM

Trackback Pings

TrackBack URL for this hissing: http://biglizards.net/mt3.36/earendiltrack.cgi/753

Comments

The following hissed in response by: Neo

Based on these two links, I'd say that the latest NSA program is quite legal, well at least if they buy them, like everybody else.

http://www.suntimes.com/output/news/cst-nws-privacy05.html

http://www.democrats.com/spy-on-them

The issue moot.

The above hissed in response by: Neo at May 15, 2006 6:15 AM

The following hissed in response by: Rovin

RovinsWorld/Conservatives for Common Sense Poll

Question: It has been reported that the Democratic Party is intending on taking control of the House and the Senate to raise your taxes, pass a national gay marriage rights bill, pull all of our troops from Iraq and Afganistan (which will create mahem and a breeding ground for a new round of terrorist), and impeach President Bush for keeping this country safe from another 9/11.

Based on what you have heard about these plans, do you support/do not support the Democratic Partys plans?

The above hissed in response by: Rovin at May 15, 2006 8:25 AM

The following hissed in response by: Terrye

Obviously it has a lot to do with the way the question is asked. Most people expect and want the government to protect them...but they are naturally reticent about their own privacy. If the program is explained to them I doubt that most people would have a problem with it. I wonder if anyone ever did a poll on Echelon? But you know what? I just don't trust most polls anymore.

The above hissed in response by: Terrye at May 15, 2006 2:45 PM

The following hissed in response by: Robert Schwartz

"More often than not, the most determinative factor in a poll's results is the actual wording of the questions themselves."

It would be more accurate to state that the only thing poll results reveal is the wording of the questions, the methods of asking them and the construction of the sample.

The above hissed in response by: Robert Schwartz at May 15, 2006 3:23 PM

The following hissed in response by: Dafydd ab Hugh

Robert Schwartz:

No, I wouldn't go that far. There is a science to polling, and properly designed polls do tell us about the full population represented by the sample.

That polls can be misused doesn't mean they're useless.

Dafydd

The above hissed in response by: Dafydd ab Hugh at May 15, 2006 4:22 PM