January 15, 2007

Polling Dos and Don'ts

Immediately after President Bush's speech on the change of course in our Iraq strategy, pollsters at both AP/Ipsos and ABC/Washington Post released surveys -- mostly conducted before Bush even spoke -- that purportedly showed huge public opposition to increasing troop levels in Iraq.

The Democrats seized upon these polls (which was the whole point) to rally Congressmen of both parties to do something, anything, to stop Bush's plan for victory before it could be implemented -- and possibly succeed.

The problem is that the very polls used as a basis of opposition by the Democrats were fundamentally flawed.

Since contemporary American pollsters -- most recently, AP/Ipsos and ABC/Washington Post polls -- evidently have a continuing problem with the basics of the science of polling, I thought I should issue a small primer (speaking as a complete non-expert in statistics and polling):

DON'T wildly overpoll members of one party, especially when asking about the policy of the other party.

The AP/Ipsos poll, to its credit, actually includes a question about party affiliation. The answer is illuminating:

Do you consider yourself a Democrat, a Republican, an Independent or none of these?

Republican ....................................... 24%

Democrat.......................................... 35%

Independent ..................................... 26%

None of these................................... 12%

Not sure.......................................... 3%

Note that the largest group among respondents is Democrat, followed by Independent; Republican is dead last; note also that the number of Democrats polled was 46% larger than the number of Republicans. It seems rather self-evident that if you poll such a huge bunch more Democrats than Republicans about a Republican policy, you're going to get a lobsidedly negative response.

Most contemporary pollsters insist this sort of question does not measure party registration, merely party identification: the purpose of this argument is to deflect criticism that they're deliberately overpolling Democrats. "No," they argue; "the Republicans' stupid policies are just causing more people to identify with the Democrats... it's really measuring a surge of support for Democrats!"

The problem with this argument is twofold:

1. If this were true, we would see such a surge in actual votes. And yet, despite the 2006 vote, no such surge is apparent: the country remains divided almost 50-50, moving sometimes left, sometimes right -- which it has been since about 1992.
2. If this were really true... then why did Ipsos shift, for this question alone, from polling "adults" to polling "registered voters?" For no other question on this poll were responses limited to registered voters.

An "adult" may well take the question to mean which party he currently likes more; but a person who is a registered voter is far more likely to take it as asking under which party he is actually registered. So in this case, I think it very likely that what we're seeing is, indeed, a huge overpolling of Democrats.

The ABC/Washington Post poll does not tell us who they polled; but considering that another question had respondents saying that they trust Democrats, by a 47% to 36% plurality over Republicans, "to do a better job handling the situation in Iraq" -- when in fact, the Democrats have never enunciated any plan at all... I think it's awfully likely that, like AP/Ipsos, they overpoll Democrats.

DO ask party affiliation -- and then DO publish the cross-tabs, so we know who is driving the response

What is the point of asking party affiliation if they're not going to bother telling us how each response broke down by party? There is a very big difference between having 70% of everyone thinking we shouldn't send more troops, and having Republicans split 50-50, Independents split 55-45, and Democrats split 99-1 (with that one being named "Joe").

DON'T poll on a technical question that respondents are simply not equipped to decide

Such as, for example, complicated strategic military questions that require several years study of military history and philosophy in order to have an informed opinion.

It's like polling people to find out whether Boeing or Lockheed Martin should build the Joint Strike Fighter... how the heck would the average American know?

DO try to include all the main points of a plan in your summary... not just those you hope will be unpopular.

In both polls, the only element of the new strategy they inquire about is the troop build-up; as AP/Ipsos phrases it:

Would you favor or oppose sending more troops to Iraq?

Surprise, surprise, when put that bluntly, 70% of respondents said "oppose," and only 26% said "favor." (In the similar ABC/WaPo poll, the response was 61% to 36% negative). But as we noted in our last post, this is actually the least important part of the change of strategic course.

It's also the most controversial... and taken in isolation, without all the other elements, even I would probably oppose it. But it should not be taken in isolation; consider this analogy:

You're a major stockholder in a company that is losing money hand over teakettle, $100 million in the last quarter alone. You determine that the problem is a VP in charge of technology who simply cannot get the new product out the door; the prototype is working great, but he's afraid to send it to production. So you go to the BoD and suggest the following:

• Sack Vice President of Technology Hammond Cheese;
• Promote his top manager, Flash Groton, who was actually in charge of the project and has been champing at the bait to send it to production;
• Accelerate testing and release of the product;
• Pour $10 million some money into advertising and promoting it, to try to recoup the company's losses.

But then, when the BoD puts it to a vote, this is how they phrase it:

'Nuff said.

DON'T precede the vital question with a series of questions designed to put people in a bad mood

What is -- or should be -- the point of this poll? To determine what Americans think about the president's new strategy -- not to see what people think of President Bush in general, or how Bush has handled Iraq up to this point, or whether the country is headed in the right or wrong direction. Those questions are of marginal interest (given that Bush is term-limited, and we'll have a new president in 2009, come hell or high ball); but this isn't the place for them.

And especially not when AP/Ipsos knows in advance that the eight questions they ask before getting to the point will prejudice the response decidedly against President Bush... just before asking the most important question about the new policy by President Bush. Regardless of what people might have thought of the policy at the beginning of the poll, by the time they've been asked these eight questions, they wouldn't support a Bush policy to encourage the Pledge of Allegiance!

Here is what they asked before troubling to inquire about "sending more troops to Iraq":

1. Is the United States on the right track or wrong track? (67% negative)
2. What's our most important problem (an entirely negative question to make people think about problems, not solutions)
3. Bush's job approval (65% negative)
4. How is Bush handling the economy? (55% negative)
5. How is he handling domestic issues? (59% negative)
6. How is he handling foreign policy and the war on terror? (60% negative)
7. How is he handling Iraq? (68% negative)
8. Congressional job approval (62% negative)
9. "Would you favor or oppose sending more troops to Iraq?" -- 70% negative.

Yes, we get it, we get it: Bush isn't very popular right now; and I have no doubt AP/Ipsos knew exactly what the response on those first eight questions would be. But what does that have to do with the change of course in our Iraq strategy?

We're changing the rules of engagement, reworking the entire Iraq strategy, and in that context, increasing our troop level to actually win this war. The efficacy and wisdom of these steps have absolutely nothing to do with Bush's popularity.

So why ask them first -- other that to sour the pool before hitting them with the real question?

Rather than honestly engage the policy itself, asking about each element of it and doing so without prejudicing the sample, AP/Ipsos chose instead to use a sleazy pollster's trick: If you want a big negative response on some message, precede it by five or six -- or eight, as in this case -- questions that will get respondents angry, depressed, and bitter about the messenger. Et voilà! Instant trashing of the message itself.

Because AP/Ipsos begins numbering the questions anew with the troop-increase question (calling it number 1 again), I thought perhaps this was a separate poll separately conducted. But the poll report only mentions the total number and type of respondents once, at the very beginning; and the dates are the same for all questions.

There is no indication that they split the sample and asked one half the political questions and the other half the "strategery" questions. And the AP write up of the poll mingles responses to both parts in the same paragraphs -- in fact, the same sentences -- making it quite clear that all questions were asked of all respondents.

The only conclusion, from a mathematical standpoint, is that this was a deliberate, cold-blooded attempt to bias the sample. The ABC/WaPo poll pulled the same trick, though to a lesser degree (and they got a less negative response -- interesting). Before asking any questions about the new policy, they first asked the following:

1. Bush's job approval (64% negative)
2. Was the Iraq war worth fighting? (58% negative)
3. Who do you trust, Republicans or Democrats, to handle the Iraq "situation"? (47% - 36% Dems)
4. Did you listen to any of Bush's speech? (58% none)
5. "Do you support or oppose Bush's proposal to send approximately 22,000 additional U.S. military forces to Iraq?"

This is not quite as bad as AP/Ipsos, as there are fewer negative questions (four instead of eight); but it's worse in the sense that ABC/WaPo makes a point of calling it "Bush's proposal," more firmly tying the strategy to the man -- after reminding respondents how much they dislike the man right now.

All in all, a shabby (and overused) trick designed, not to probe the public's response, but to push it firmly against the new strategy.

DO construct a poll that elicits real information; DON'T release a divisive push-poll in the middle of a war

The very science of polling is under fire today in America; and this kind of polling is one big reason why. Most people understand exactly what Mark Twain meant when he wrote:

We all know instinctively that pollsters can manipulate questions and reponses to get whatever answers they (or more accurately, their clients) desire. But the discrediting of polling in American politics stems not from the knowledge that political pollsters can manipulate polls -- but rather from the deep suspicion that they do manipulate them, nearly every time.

Similarly, we don't distrust the media because it's possible for them to lie, but because most of us believe they lie like a Persian rug every blessed day. And why, in both cases? Because liberals believe they, the "Anointed," are so much smarter than we, that we simply must be led, like children, for our own good... otherwise, we might draw the wrong conclusions, acting against "the Vision," thus falling into mortal sin.

It is utterly clear that the editors and publishers at the Associated Press, the Washington Post, and ABC, along with the pollsters and polling directors, share belief in a number of leitmotifs about the Iraq war:

• Iraq has nothing to do with the larger War Against Global Jihad;
• Terrorism is not a major threat to the United States;
• Bush and the GOP suffer from "Islamophobia;"
• We went to war in Iraq to steal their oil;
• We failed to steal any oil, therefore we've already been defeated;
• We must admit defeat and get out, having learned a hard lesson about trying to steal other people's oil;
• "Changing course" can only mean withdrawal from Iraq; adding troops, no matter what other strategic changes we make, constitutes "staying the course" -- which has already been discredited;
• If enough Americans demand that we get out, Bush will have to comply;
• The most urgent goal is for Democrats to win the presidency: all other goals, including national security, take a back seat.

They see polliing, not as a scientific or mathematical tool to understand the electorate (or the citoyens) -- what they think, what they want -- but as a political tool to shape and mold public opinion to support Democratic and liberal goals and initiatives. As we have noted many times before, if these errors and mistakes in polling were random, they would favor the Right as often as they favor the Left. Instead, they always line up to promote Democratic policies and damage Republican policies (the Restaurant Check Fallacy: errors in toting up the check always favor the restaurant, never the customer).

Rather than turning polling into the scientific-political tool the Left wants, however, they have only succeeded in tainting all polling as disreputable -- today, even good polling must scale a wall of incredulity to be heard. That is the natural outcome when ordinary people -- who may not have specialized training but are much smarter than the elites think they are -- realize how they have been bamboozled and beguiled by polling in the past.

This is a sad turn of affairs, and it will take many years to undo. But the healing can only begin when the self-inflicted injuries cease.

Shame on AP/Ipsos and ABC/Washington Post pollsters, for allowing liberal politicos to make fools of them.

Hatched by Dafydd on this day, January 15, 2007, at the time of 5:05 PM

Trackback Pings

TrackBack URL for this hissing: http://biglizards.net/mt3.36/earendiltrack.cgi/1665

Comments

The following hissed in response by: Terrye

The above hissed in response by: Terrye at January 16, 2007 3:52 AM

The following hissed in response by: Steelhand

The above hissed in response by: Steelhand at January 16, 2007 4:16 AM

The following hissed in response by: snochasr

The above hissed in response by: snochasr at January 16, 2007 6:18 AM

The following hissed in response by: Freetime

The above hissed in response by: Freetime at January 16, 2007 8:21 AM

The following hissed in response by: Big D

The above hissed in response by: Big D at January 16, 2007 9:34 AM

The following hissed in response by: SkyWatch

The above hissed in response by: SkyWatch at January 16, 2007 11:31 AM

The following hissed in response by: Big D

The above hissed in response by: Big D at January 16, 2007 3:30 PM

Post a comment

© 2005-2009 by Dafydd ab Hugh - All Rights Reserved