Readers have asked me to demonstrate how I go about reading a study. We don’t use privately-funded studies as examples, so I’ve been waiting for something interesting to show up in the public domain where we all have free access to the information.

A little background.

Postgraduate training in the quantitative behavioral sciences devotes years to the nuts and bolts of research design, sampling, methods, statistics, and analytics. It also teaches hard-earned lessons in objectivity, discipline, and rigor.

In one of my classes at Indiana University the weekly assignment was to write a five page paper summarizing 500 pages of scientific readings. The first week I wrote what I considered to be an excellent paper. It came back with this scribbled in red ink on page one: It is of no interest to me what you think. 

Ouch! My error was in confusing my opinion with actual facts. 

All research clients want the facts. When they also want our opinions, we tell project directors to follow these rules: 

• Our opinions always go in a separate chapter.
• Our opinions always come after we have presented the facts.
• We always clearly label our opinions as subjective points of view.

The difference between how pros and amateurs read studies.

Pros begin with a thorough examination of the methods. They know if the tools and processes used to conduct the research don’t meet scientific standards, the findings are worse than irrelevant. Pros have learned to read this section first so when they recognize the earmarks of sloppy science, they don’t need to waste their time reading about it.

Which is worse, bad research or none at all? 

Every person in every class I ever taught gets it wrong. Bad research causes two insurmountable problems: it points us in the wrong direction and it gives us a false sense of confidence we’re going the right way. Let’s say you ask me which way to the airport and I send you in the wrong direction. Off you go, blissfully ignorant, until you go far enough down the wrong road to finally give up, turn around, and go the other way.

So up jumps this from the International Journal of Aviation, Aeronautics, and Aerospace. 

A new study says peoples’ preference for driverless cars will wreak havoc with the entire airline industry, from airlines scheduling fewer daily flights to airport hotels losing customers to the airline manufacturers selling fewer airliners. 

It seemed quite a claim, so we took a closer look at the evidence.

In the very first paragraph of the introduction to their report, Embry-Riddle Aeronautical University professors Stephen Rice and Scott Winter claim “the effects could very well be catastrophic.”  

What kind of catastrophe are we talking about here?

Rice and Winter predict fewer passengers, fewer flights and fewer airplanes. According to them, the entire commercial air travel industry is destined to shrink drastically.

Regular readers know of my commitment to seeking alternate sources of information. 

It took me one minute to find these two bits of data: 

• The International Air Transport Association says 7 billion passengers will fly commercial in 2035, nearly double the number today. 
• PwC’s annual report on the state of the airline industry says the demand for air travel will double in the next fifteen years. 

So the IATA and PwC say the industry will double and this new study predicts fewer passengers, fewer flights and fewer airplanes. Which makes more sense to you?

The authors are well-qualified experts. 

Their published work includes A design‐by‐treatment interaction model for network meta‐analysis with random inconsistency effects; A test of the null hypothesis significance testing procedure correlation argument; and Consumer perceptions about cabin depressurization during hijackings.

I was not able to find any detailed information about methods or sample.

All we know is that it was a sample of 2,000 and it skewed young. My requests for details went unanswered, so we don’t know who, where, or how. Wouldn’t you like to know how different groups compare, say for example, vacation travelers vs business travelers? I would. And don’t people’s opinions vary by age, income, education, occupation, and so on? Studies don’t have much value when you can’t compare and contrast different categories.

When people go looking for information to support what they already believe, they find it.

Researchers should know better, but they don’t always. Most academic researchers bend over backwards to not conclude anything at all, obsessed as they are by the holy grail of failing to reject the null hypothesis. When people begin with the answers, they design their research in ways that virtually guarantee the results they want. This study tilted the playing field by choosing an approach that presented carefully scripted scenarios to study subjects. 

If this was a courtroom, they’d be guilty of leading the witness.

Study subjects “led witnesses” with scripts that highlighted the pleasures of riding in robo-cars while emphasizing the pains of flying on airplanes. They prefaced the presentation of their scenarios by reminding study subjects how air travelers hate security checks, long lines, delays, and risks of losing baggage. They also promoted the advantages of riding in a driverless car, telling subjects they could relax, sleep, watch a movie, etc. A balanced comparison would have listed advantages and disadvantages of both, or of neither, but not the pros of one and the cons of the other. The researchers stacked the deck from both ends.

Another fundamental flaw is asking people what they would do in a hypothetical situation. 

Every bit of good research I have ever read says people are really lousy at predicting the future. Why? Because our future behaviors are situation-specific and are determined by the conditions at the time  – things we cannot possibly know in advance.

And it’s not simply riding in a driverless car, either. 

Current day examples of driverless cars are mostly cars puttering around town at low speeds going only short distances with a safety driver behind the wheel. Riding long distances for hours and hours at high speeds in a driverless car across unfamiliar terrain is a completely different situation, isn’t it?

Here is the first of their librettos, presented verbatim. 

Imagine a scenario where you live in Fairfax, Virginia, which is a suburb of Washington DC. You wish to travel to New York City to see the Empire State Building. You are traveling alone. You plan to arrive there at 3:00 PM the following day. To take this trip via commercial airline, you would need to arrive at a New York airport by 2:00 PM to catch a taxi to get you to the specific location. This means departing from Dulles or Reagan National around 1:00 PM. You need to arrive at the airport 2 hours before your flight, which would be 11:00 AM. You must leave your house 30-60 minutes before that to ensure that you make the flight on time. In summary, your total travel time is from 10:00 AM to 3:00 PM, or 5 hours. Google Maps indicates that you can make the drive in 4 hours 46 minutes in average traffic. In other words, you can choose between two options with almost identical travel times. Which would you prefer?

The authors say, “When presented with this scenario, 33.67% said they preferred to fly commercial.”

If you’re like me, you are stunned by this finding. One in three choosing flying means two out of three study subjects said they’d rather drive from Fairfax, VA to NY, NY than fly. 

Let’s go back and look at that scenario again. 

Researchers used 99 words to describe the trip by air, including three admonishments: “you need to,” “you need to,” and “you must.” They used only seventeen words to report the drive time and included no such exhortations. Why is this important? Subconsciously, respondents have just learned that traveling by air is five times as difficult as driving. When you first read the story they made up, did you notice study subjects were given the choice between taking a 5-hour commercial flight or driving themselves for 250 miles? I didn’t. Confused by the references later and thinking the choice was between driverless cars and flying, I had to go back and check.

What study subjects were told next.

Now imagine that you have a driverless car that can navigate you to your destination with no input whatsoever from the driver. In fact, you can lie down and sleep or watch a movie while your driverless car transports you to New York City. Now, which choice would you prefer? 

The authors reported 33.27% of the total respondents said they would prefer to fly commercial. If you’re like me, you are even more stunned by this one. Only four-tenths of one percent more people chose robo-taxis over driving themselves (33.67% – 33.27%) If driverless cars are going to have catastrophic effects on the entire airline industry, won’t they need to have an impact greater than four tenths of one percent over the current drive ourselves situation?(This statistic completely undermines their pre-conclusions, which probably explains why we never see this little nugget again.)

Researchers go on to present more scenarios where the drive time increases to 7, then 11, 21, and 45 hours.

Generally speaking, the longer the drive, the less likely people want to go by car, driverless or otherwise. Data from this study show one person in six would rather spend 45 hours in a car than fly commercial. Does that statistic shock you? It did me.

The scenarios say nothing about traffic jams, accidents, flat tires, or the inevitable road construction delays. The choice itself doesn’t take enough things into account. Travel decisions include paying for the trip. Forcing an artificial choice that does not include trip costs is ignoring what we know to be a primary factor in travel decision-making. Eliminating cost from the equation invalidates study findings. They added a proviso later that said you’d have to rent a car at your destination if you flew. This resulted in even more people saying they’d take a driverless car than fly. This is shamelessly stacking the deck because in their envisioned world of robo-cars, you would never have a need to rent a car, because there would be driverless cars when you got there.  

There is not a shred of doubt in my mind that as robo-cars become commonplace, more travelers will choose them for short trips. 

But, no, no, no, I do not believe the evidence these scholars present should be interpreted to mean the entire commercial air travel industry is doomed. That’s only how I read it, though. Dozens of media outlets picked up this story and passed it along unquestioned. A more accurate title for this story would be:” Despite every effort to influence the outcome so as to provide evidence that proves what we already believe, fewer than one half of one percent of travelers report imagining themselves preferring robo-cars to driving themselves instead of flying commercial.”

The authors close by saying the airline industry and the FAA have not given this problem much thought, the problem is not nearly discussed as it needs to be, and it is time to start thinking about the problem “before the hammer hits.” 

There is one more thing you should know about academic research.

It is typically funded by grants from the government. If you can establish yourself as an authority on a given topic, more funding will come your way. Which reminds me of this 9-second scene from Blazing Saddles.

More of what passes for research is bad than good. How good are you at telling the difference?  How good would you like to be? I can teach you how to see through manipulations, distortions and fabrications.

Take a Closer Look, Volume 2, is free to Kindle Unlimited customers. The best way to protect yourself against the manipulations, distortions and fabrications that are more and more prevalent these days is to learn how to see through them.