Readers have asked me to demonstrate how I go about reading a study. We don’t use privately-funded studies as examples, so I’ve been waiting for something interesting to show up in the public domain where we all have free access to the information.

A little background

Postgraduate training in the quantitative behavioral sciences devotes years to the nuts and bolts of research design, sampling, methods, statistics, and analytics. It also teaches hard-earned lessons in objectivity, discipline, and rigor.

In one of my classes at Indiana University the weekly assignment was to write a five page paper summarizing 500 pages of scientific readings. The first week I wrote what I considered to be an excellent paper. It came back with this scribbled in red ink on page one: It is of no interest to me what you think. 

Ouch! My error was in confusing my opinion with actual facts

All research clients want the facts. When they also want our opinions, we tell project directors to follow these rules: 

• Our opinions always go in a separate chapter.
• Our opinions always come after we have presented the facts.
• We always clearly label our opinions as subjective points of view.

The difference between how pros and amateurs read studies

Pros begin with a thorough examination of the methods. They know if the tools and processes used to conduct the research don’t meet scientific standards, then the findings are worse than irrelevant. Pros have learned to read this section first so when they recognize the earmarks of sloppy science, they don’t need to waste their time reading about it.

Which is worse, bad research or none at all? 

Every person in every class I ever taught gets it wrong. Bad research causes two insurmountable problems: it points us in the wrong direction and it gives us a false sense of confidence that we’re going the right way. If you ask someone which way to the airport and they send you in the wrong direction, then off you go, blissfully ignorant, until you go far enough down the wrong road to finally give up, turn around, and go the back in the other direction

So up jumps this from the International Journal of Aviation, Aeronautics, and Aerospace 

A new study says peoples’ preference for driverless cars will wreak havoc with the entire airline industry, from airlines scheduling fewer daily flights to airport hotels losing customers to the airline manufacturers selling fewer airliners. 

It seemed quite a claim, so I took a closer look at the evidence

In the very first paragraph of the introduction to their report, Embry-Riddle Aeronautical University professors Stephen Rice and Scott Winter claim “the effects could very well be catastrophic.”  

What kind of catastrophe are we talking about here?

Rice and Winter predict fewer passengers, fewer flights, and fewer airplanes. According to them, the entire commercial air travel industry is destined to shrink drastically.

Regular readers know of my commitment to seeking alternate sources of information 

It took me one minute to find these two bits of data: 

• The International Air Transport Association says 7 billion passengers will fly commercial in 2035, nearly double the number today. 
• PwC’s annual report on the state of the airline industry says the demand for air travel will double in the next fifteen years. 

So the IATA and PwC say the industry will double and this new study predicts fewer passengers, fewer flights, and fewer airplanes. Which makes more sense to you?

The authors are well-qualified experts 

Their published work includes “A design‐by‐treatment interaction model for network meta‐analysis with random inconsistency effects; a test of the null hypothesis significance testing procedure correlation argument; and consumer perceptions about cabin depressurization during hijackings.” With catchy titles like that, who wouldn’t want to dive into that study?

Embry-Riddle Aeronautical University (erau.edu)

I was not able to find any detailed information about methods or sample

All we know is that it was a sample of 2,000 and it skewed young. My requests for details went unanswered, so we don’t know who, where, or how. 

Wouldn’t you like to know how different groups compare, say for example, vacation travelers vs business travelers?

I would. And don’t people’s opinions vary by age, income, education, occupation, and so on? Studies don’t have very much value when you can’t compare and contrast different categories.

When people go looking for information to support what they already believe, they find it

Researchers should know better, but they don’t always. Most academic researchers bend over backwards to not conclude anything at all, obsessed as they are by the holy grail of failing to reject the null hypothesis. When researchers begin with the answers, they design their studies in ways that virtually guarantee the results their clients are hoping for. The “Airlines Doomed” study tilted the playing field by choosing an approach that presented carefully scripted scenarios to study subjects. 

If this was a courtroom, they’d be guilty of leading the witness

Study subjects “led witnesses” with scripts that highlighted the pleasures of riding in robo-cars while emphasizing the pains of flying on airplanes. They prefaced the presentation of their scenarios by reminding study subjects how air travelers hate security checks, long lines, delays, and risks of losing baggage. This guarantees the study subjects will be negatively predisposed to  air travel.

The study designers also promoted the advantages of riding in a driverless car, telling subjects they could relax, sleep, watch a movie, etc. This guarantees the study subjects will be positively predisposed to riding in a driverless car. 

A balanced comparison would have listed advantages and disadvantages of both so the playing field is level, but never the pros of one and the cons of the other. These researchers stacked the deck from both ends.

Another fundamental flaw is asking people what they would do in a hypothetical situation 

Every bit of good research I ever read says people are really lousy at predicting the future. Why? Because our future behaviors are situation-specific and are determined by the conditions at the time  — things we cannot possibly know in advance.

And it’s not simply riding in a driverless car, either 

Current day examples of driverless cars are mostly cars puttering around town at low speeds going only short distances with a safety driver behind the wheel. Riding long distances for hours and hours at high speeds in a driverless car across unfamiliar terrain is a completely different situation isn’t it?

Here is the first of their librettos, presented verbatim 

Imagine a scenario where you live in Fairfax, Virginia, which is a suburb of Washington DC. You wish to travel to New York City to see the Empire State Building. You are traveling alone. You plan to arrive there at 3:00 PM the following day. To take this trip via commercial airline, you would need to arrive at a New York airport by 2:00 PM to catch a taxi to get you to the specific location. This means departing from Dulles or Reagan National around 1:00 PM. You need to arrive at the airport 2 hours before your flight, which would be 11:00 AM. You must leave your house 30-60 minutes before that to ensure that you make the flight on time. In summary, your total travel time is from 10:00 AM to 3:00 PM, or 5 hours. Google Maps indicates that you can make the drive in 4 hours 46 minutes in average traffic. In other words, you can choose between two options with almost identical travel times. Which would you prefer?

The authors say, “When presented with this scenario, 33.67% said they preferred to fly commercial”

If you’re like me, you are stunned by this finding. One in three choosing flying means two out of three study subjects said they’d rather drive from Fairfax, VA to NY, NY than fly. 

Let’s go back and look at that scenario again 

• Did you see where the researchers used 99 words to describe the trip by air only seventeen words to report the drive time?
• Did you see where the researcher included three warnings in their description of the air trip (You need to, You need to, You must) and none about driving?

Why is this important?

Subconsciously, respondents have just had it explained to them that traveling by air is five times as difficult as driving. When you first read the story they made up, did you notice study subjects were given the choice between taking a 5-hour commercial flight or driving themselves for 250 miles? I didn’t. Confused by the references later and thinking the choice was between driverless cars and flying, I had to go back and check.

What study subjects were told next

Now imagine that you have a driverless car that can navigate you to your destination with no input whatsoever from the driver. In fact, you can lie down and sleep or watch a movie while your driverless car transports you to New York City. Now, which choice would you prefer? 

The authors reported 33.27% of the total respondents said they would prefer to fly commercial. If you’re like me, you are even more stunned by this one. Only four-tenths of one percent more people chose robo-taxis over driving themselves (33.67% – 33.27%).

If driverless cars are going to have catastrophic effects on the entire airline industry, won’t they need to have an impact greater than four tenths of one percent over the current drive ourselves situation? This statistic completely undermines their pre-conclusions, which probably explains why we never see this little nugget again.

Researchers go on to present more scenarios where the drive time increases to 7, then 11, 21, and 45 hours

Generally speaking, the longer the drive, the less likely people want to go by car, driverless or otherwise. Data from this study show one person in six would rather spend 45 hours in a car than fly commercial. Does that statistic shock you? It did me. Maybe it’s someone so afraid of flying that they won’t get on an airplane no matter what.

Did you notice how scenarios say nothing about traffic jams, accidents, flat tires, or the inevitable road construction delays?

The choice between the two — as presented — doesn’t take enough things into account. Travel decisions include paying for the trip. Forcing an artificial choice that does not include trip costs is ignoring what we know to be a primary factor in travel decision-making. Eliminating cost from the equation invalidates study findings. They added a proviso later that said you’d have to rent a car at your destination if you flew. This resulted in even more people saying they’d take a driverless car than fly. This is shamelessly stacking the deck because in their envisioned world of robo-cars, you would never have a need to rent a car, because there would be driverless cars when you got there.  

There is not a shred of doubt in my mind that as robo-cars become commonplace, more travelers will choose them for short trips

But, no, no, no, I do not believe the evidence these scholars present should be interpreted to mean the entire commercial air travel industry is doomed. That’s how a well-trained and experienced pro (me) read it.

Dozens of media outlets picked up this story and passed it along unquestioned

A more accurate title for this story would be:” Despite every effort to influence the outcome so as to provide evidence that proves what we already believe, fewer than one half of one percent of travelers report imagining themselves preferring robo-cars to driving themselves instead of flying commercial.”

The authors close by saying the airline industry and the FAA have not given this problem much thought, the problem is not nearly discussed as it needs to be, and it is time to start thinking about the problem “before the hammer hits.” 

There is one more thing you should know about academic research

It is typically funded by grants from the government. If you can establish yourself as an authority on a given topic, more funding will come your way. Which reminds me of this 9-second scene from Blazing Saddles.

Bonus

Librettos are the text part of operas.