Early Tuesday night, the NY Times said Hillary Clinton was the overwhelming favorite to win the presidential election. Wednesday morning this false conclusion sounded like the most famous election headline of all time.
The problems with political polling are the same as the problems with any kind of surveying. Data scientists know every survey can go wrong in three different places:
- How the data are collected.
- How the data are corrected.
- How the data are interpreted.
Most surveys and polls are flawed on at least one dimension.
Many are flawed on two, and some, like the 2016 U.S. Presidential election, on all three.
1. Pressured to provide information quickly and cheaply, pollsters and surveyors take shortcuts and cut corners when they collect the data. These errors are mostly mechanical.
2. Once the often-flawed data have been collected, statisticians “correct” the numbers by a process they call weighting, where responses are subjectively adjusted to conform to a set of expectations. Think of it as the voices of subgroups (students, women, retirees, etc.) being amplified or muted. There is much evidence that weighting schemes are often deeply flawed, magnifying some findings and diminishing others. These mistakes in judgment are grounded in the assumptions study sponsors are willing to make.
3. Interpretations are subjective ways of explaining things and are the most personal of these three sources of error. As every psychologist knows, human beings generally make up their minds without carefully considering alternatives. Then two other things happen. They ignore evidence that conflicts with the positions they’ve taken, and they seek out evidence that confirms and supports their beliefs.
Reporters believed Clinton would win, so they conducted their polls and interpreted their results just that way.
Political pollsters aren’t the only culprits, though.
Businesses often deliberately inflate or deflate findings to fit their preferred narratives, a situation repeatedly played out in executive suites and boardrooms. When they conduct research, businesses cut corners, massage the data, and interpret things in the most flattering ways they can. And then they wonder why customers aren’t “voting” the ways decision-makers hoped they would.