Adventures in False Precision Part III, Those Funny Polls

A while back I talked about how scientists try to make precise quantitative predictions that just aren't theoretically possible. They do it because of public need. The public needs forecasts. It doesn't really matter if they are accurate or not. It comforts the public to see those numbers there.

The need for the public to possess forecasts no matter how bogus runs across the social and scientific sphere. From hurricane landfall predictions to earthquakes to Federal budget deficit predictions, very educated people produce forecasts that have no real scientific basis. In a world where people routinely consult horoscopes, I guess it isn't so surprising that we demand and get scientists and economists to try and do the impossible.

So it goes with political polling. The public wants to know if a certain candidate is going to win. They want to know the bottom line in advance. And in fact, obtaining that bottom line isn't at all possible. But we hire pollsters to try and do it anyway.

Pollsters will contact a few hundred to a thousand people and ask them who they will vote for in an upcoming election. They do it by phone. Many of the people making these calls for the pollsters are not particularly well trained to ask questions. The calls they make are on land lines only and many people refuse to answer. So we have some significant problems from the get go. There is some inherent bias.

The pollster isn't actually sampling a full set of the population. Rather it's those willing to listen to a droning, halting voice of some pimple faced kid working for minimum wage and answer his questions. How many people are willing to do this? The pollster can only reach those with a land line (forget about the cell phone generation) and those polite or docile enough to answer his or her questions. I know for instance that I never answer these questions. I just hang up.

Then there is another problem that's fundamental. A person is asked who they will vote for. But they are nowhere near a polling booth. There is time and distance between when they utter "I'm going to vote for Sid Caesar" and if and when they actually go to the polls. What they are often really saying when they answer the question is "I think I'm going to vote for Sid Caesar" or "I would like to vote for Sid Caesar" or "I'm too embarrassed to say that I don't like Sid Caesar because he has a big nose so I'll say I'll vote for him anyway." A good percentage of people who say they will vote for someone when called by phone undoubtedly vote for someone else or don't vote at all.

The inability to randomly sample people and the difference between what people say they are going to do and what they actually do in a voting booth introduce large errors in predictions. But if a pollster actually included those errors, their answers to "who is going to win" would be too vague. The bottom line is that most elections are decided by less than a 10 percentage difference between two candidates. The pollster knows this intuitively. The pollster also knows he or she can't simply open up and state the facts. For instance if the pollster said the following, no more jobs would be forthcoming:

"Sid Caesar is ahead by eight points in our poll, but my error in forecasting is so large that he could actually lose. I can't really say who is going to win or lose, sorry."

Faced with such a potential problem in credibility, the pollster - maybe unconsciously, maybe on purpose - systemically and significantly underreports errors in the forecasts. Typically, they will state errors of at most two percentage points. This systematic underreporting of errors comforts the public and gives the forecasts an air of legitimacy. The news services report these results as if they bear some relation to reality. The polls become news. We watch the trends unfold and get excited as one person moves ahead or behind in the polls. But in fact, all of this movement is simply noise. The trends are nonsense and the forecasts are nonsense.

How much error is there actually in these election forecasts? A lot more than the pollsters indicate. For example, let's look at the recent primary election in New Hampshire. Polls had Obama surging to a 10 to 12 point lead prior to the election. In fact he lost by 2 points. This was not an isolated or unusual occurrence. A month later much the same thing happened in California. Polls had Obama up slightly prior to the election. In fact he lost by 10 points.

For those who might think that the polls are just wrong when it comes to forecasting Obama victories, I note that they indicated that Huckabee would finish third in Georgia with 28 percent of the vote. In fact he won with 34 percent.

The proof is in the pudding. Errors in polls by as much as 14 percent are not unusual. Since most elections are won by amounts far smaller than this number, most polls are in fact worthless for prediction. They can predict the obvious case when a candidate is ahead by a wide margin. But if the election is tight, a candidate is ahead by 55 to 45 or less, they can't do the job.

Why do we listen to forecasts from pollsters when in hindsight they can't make the predictions they say they can? My guess is that it's a psychological thing. It gives us comfort to think we know the unknowable. Like when we consult an astrologer, we fool ourselves into thinking these pollsters actually know what they are doing. Plus it's good water cooler gossip to watch a candidate rise and fall in the polls. Ultimately, though, the pollsters are providing not facts but carnival, PT Barnum style entertainment.