October 4, 2010

The Loudest Minute Defended

Hatched by Dafydd

Commenter MikeR asked in comments to the first installation of this mini-series of posts, the Loudest Minute, whether I had seen the related posts at the New York Times blog FiveThirtyEight, written by former Daily Kos blogger and statistician Nate Silver... posts that (not surprisingly) extolled the accuracy of polling.

I seem to have misunderstood MikeR's point, which was merely to draw the posts to my attention. See, at first glance, the FiveThirtyEight posts appear to contradict my back-of-the-pants analysis of races in which the well-known incumbent is unable to rise into a comfortable majority in the polls, despite being up against a much more obscure opponent. Thus I mistakenly thought MikeR wanted me to "square" my analysis with that of FiveThirtyEight, when he was only curious whether I'd seen them.

But since squaring that circle makes a more compelling post than simply writing, "No, I hadn't read them until now," I shall continue hence as originally posted.

It's easy to square my analysis with that of FiveThirtyEight because we're not in conflict: Not even Nate Silver says polls are always accurate... just that they're generally accurate.

Let's look at his table of accuracy, the one he introduces in The Uncanny Accuracy of Polling Averages*, Part II (I'll only look at Senate races, for illustrative purposes).

His database includes all elections that took place on normal November election dates since 1998, in which multiple pollsters produced polls about thirty days out from the election; in this case, we're looking at 76 Senate elections:

Performance of (Senate) candidates with lead in simple polling average 30 days before election
Polling lead Number of races Won - lost Win percent
0 - 3 points 15 8 - 7 53%
3 - 6 points 12 9 - 3 75%
6 - 9 points 7 7 - 0 100%
9 - 12 points 9 9 - 0 100%
12 - 15 points 8 8 - 0 100%
15+ points 42 42 - 0 100%

(Note that the percentages in my table differ from those in Silver's, because I'm only looking at Senate races.)

Let's look at the three races I cited as examples where the candidate currently behind will likely win, because the better-known candidate has a small lead, but remains mired below or right at 50% (thus is "unable to close the deal" with voters).

  • Senate Majority Leader Harry "Pinky" Reid (D-Caesar's Palace, 95%), currently ahead by 1.4 points in the RCP average.
  • Sen. Patty Murray (D-WA, 95%), currently ahead by 3.3 points in the RCP average.
  • Sen. Barbara Boxer (D-CA, 100%), currently ahead by 6.2 points in the RCP average.

According to Silver's own chart, Nevada (Reid) is in the category where 53% of leading candidates won their races; Washington state (Murray) is in the category where 75% of leading candidates won; and California (Boxer) is in the category where 100% of leading candidates won.

Let's leave the California race for last. In the first two, the lead does not create a particularly daunting challenge: Silver himself would say that the Nevada and Washington races were reasonably likely to go to the challenger, because historically, a reasonably large number did just that. Therefore, in two of my three examples, the "null hypothesis" is not ruled out; there is no discrepency between what I wrote and what Nate Silver wrote, therefore nothing to explain, justify, or square.

So let's turn to the one exception, the California race between Barbara Boxer and Carly Fiornina. In that race, Boxer is 6.2% ahead in the RCP average, landing in the category that has a 100% reelection rate since 1998. Does that mean Boxer is destined to hang onto her seat?

No, because it's not as simple as that. Admittedly, the California election is dicier than the other two; but let's turn to Mr. Silver again:

Mr. Toomey, for instance, is regarded as a 92 percent favorite by our model, which corresponds quite nicely to the 89 percent winning percentage that I described above. His winning percentage is a tiny bit higher than it might be for another candidate with a similar lead in the polls, because some of the other factors we account for in our model. For instance, there are an especially large number of polls in Pennsylvania, and they are all quite consistent with one another, which speaks toward his lead being slightly more robust than usual. In other cases -- if the polling is sparse or inconsistent, or if an unusually large number of undecided voters remain in the race -- the model will increase the uncertainty it attaches to a forecast.

In the Boxer-Fiorina race, the incumbent is 6.2 points up; but there is a relatively large 11.2% undecided, nearly double Boxer's lead. Fiorina can win the race by capturing 78% of the undecided vote, without having to flip a single Boxer supporter.

Too, the polls in the California race are not "consistent with one another;" the polls in the month of September range from Fiorina up 2 to Boxer up 9, an 11-point variance. By contrast, in the Pat Toomey race in Pennsylvania (which Silver cites as an example and stepping-off point for his post), the September polls range from Toomey up 3 to Toomey up 9, only a six-point variance, just over half that of the California race.

The most recent CNN/Time poll in California came in very high for Boxer; it's the highest lead any non-partisan poll has given her since May, and it's likely an outlier. By contrast, the most recent poll, SurveyUSA, gives Boxer a lead of only one-third the CNN poll.

If the next CNN poll comes out with Boxer up 4 instead of 9 (which is what the previous CNN in early September found), then Boxer would only be ahead by an RCP average of 5.3 points -- which would put the race in the same category as the Washington state Senate race. In that category, three out of 12 elections in Silver's database went to the underdog in the poll. Only a single poll -- a likely outlier at that -- puts the California race into the "6 to 9 point" category... so I give it less credence.

In any event, we'll see fairly soon; I stand by my prediction that all three Republicans will win.

Hatched by Dafydd on this day, October 4, 2010, at the time of 7:10 PM

Trackback Pings

TrackBack URL for this hissing: http://biglizards.net/mt3.36/earendiltrack.cgi/4618


The following hissed in response by: MikeR

I appreciate this post a lot. I do deny, however, that I asked you "how I squared my back-of-the-pants analysis of races ... with the series of blogposts on the New York Times blog FiveThirtyEight." I asked only about what you thought of the series of blogposts there, because I thought they were interesting.

The above hissed in response by: MikeR [TypeKey Profile Page] at October 5, 2010 1:51 PM

The following hissed in response by: Dafydd ab Hugh


Sorry, I must have misunderstood: At first glance, the posts at FiveThirtyEight appear to contradict my predictions; and I thought your post was meant to bring them to my attention so I could address the seeming dissent. I'll rewrite the first sentence of this post.


The above hissed in response by: Dafydd ab Hugh [TypeKey Profile Page] at October 6, 2010 3:53 AM

The following hissed in response by: MikeR

Also interesting from Nate Silver:

The above hissed in response by: MikeR [TypeKey Profile Page] at October 11, 2010 10:18 AM

The following hissed in response by: Dafydd ab Hugh


Thanks... this might be an occasion for another post!


The above hissed in response by: Dafydd ab Hugh [TypeKey Profile Page] at October 11, 2010 2:21 PM

Post a comment

Thanks for hissing in, . Now you can slither in with a comment, o wise. (sign out)

(If you haven't hissed a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Hang loose; don't shed your skin!)

Remember me unto the end of days?

© 2005-2009 by Dafydd ab Hugh - All Rights Reserved