Early Tuesday night, the New York Times said Hillary Clinton was the overwhelming favorite to win the presidential election. Wednesday morning this false conclusion sounded like the most famous election headline of all time
The problems with political polling are the same as the problems with any kind of surveying
Data scientists know there are three ways every survey can go wrong:
- How the data are collected.
- How the data are corrected.
- How the data are interpreted.
Most surveys and polls are flawed on at least one dimension
Many are flawed on two, and some, like the 2016 U.S. Presidential election polls, on all three.
1. Pressured to provide information quickly and cheaply, pollsters and surveyors take shortcuts and cut corners when they collect the data. These errors are mostly mechanical.
2. Once the often-flawed data have been collected, statisticians “correct” the numbers by a process they call weighting, where responses are subjectively adjusted to conform to a set of expectations. Think of it as the voices of subgroups (students, women, retirees, etc.) being amplified or muted. There is much evidence that weighting schemes are often deeply flawed, magnifying some findings and diminishing others. These mistakes in judgment are grounded in the assumptions study sponsors are willing to make.
3. Interpretations are subjective ways of explaining things and are the most personal of these three sources of error. As every psychologist knows, human beings generally make up their minds without carefully considering alternatives. Then two other things happen. They ignore evidence that conflicts with the positions they’ve taken, and they seek out evidence that confirms and supports their beliefs.
Reporters believed Clinton would win, so they conducted their polls and interpreted their results just that way
Political posters aren’t the only ones
Businesses often deliberately inflate or deflate findings to fit their preferred narratives, a situation repeatedly played out in executive suites and boardrooms. When marketers conduct research, they cut corners, massage the data, and interpret things in the most flattering ways they can. And then they wonder why customers aren’t “voting” the ways decision-makers hoped they would.
The best way to interpret political polls is to add them all together and take an average.
