How Did Businesses Waste $3 Billion on Surveys?

It was early evening at the garden party celebrating the couple’s magnificent new backyard deck. Built into the gentle slope of the land were walkways, conversation areas, and bridges spanning koi ponds, all made from materials of the highest quality. Gentle breezes sighed in the willows, a jazz trio was very cool on the veranda, lanterns swayed in the trees and everywhere the scent of night-blooming jasmine. White-gloved waiters glided about serving champagne and canapés on gleaming silver trays. A guest, marveling at the beauty of it all, commented that it must have involved an enormous amount of planning and craftsmanship. The hostess replied with a dismissive wave of her bejeweled hand, “No, it’s quite easy, actually. You just get some boards and a husband.”

It’s easy to write a survey, too

Anyone can do it. They do it every day on Family Feud. They survey 100 members of a studio audience. There is no attempt to be scientific, because it’s only a game show and it’s all in fun.

Writing a good survey is a different matter

The only way we can collect accurate and objective data is to rigorously adhere to ALL the principles universities teach in their quantitative behavior programs, not just one or two. When it comes to fundamental truths, you can’t adhere to ‘em if you’re not aware of ‘em.

When a world-renowned medical researcher came to our small shop and asked us to write a survey, I was flabbergasted

After all, he was a big shot and we were very small potatoes. He told me the data he collected would provide the raw materials for half a dozen articles he would publish in prestigious academic journals. He reasoned that he wrote only one questionnaire every five or six years and we wrote hundreds, so he would be a fool to write his own. A rare client, indeed.

The general decline of research standards and practices

Here are five things contributing to the decrease in quality of the research we see, particularly the survey research:

Research departments are buried deeply in most organizational hierarchies. Projects go down chutes and up ladders, processed by gatekeepers who don’t know how to tell good surveys from bad and don’t often care, either.
Research has been made subordinate to marketing. This is fundamentally flawed because marketing wants to influence people and research wants to understand them. Conflict is inevitable, and guess what happens?
Marketing-led research cares little about rigor. When you have the Count Chocula account, you proudly cite bought-and-paid-for research that shows it is “part of a balanced diet.”
Most clients demand faster and cheaper surveys. Research firms take more shortcuts and cut more corners.
Writing your own surveys has never been more popular.

DIY survey sites contain pre-emptive disclaimers warning users that online surveys have to be clear, concise, and unbiased

Good advice, to be sure, but we can’t help but notice they are telling people what to do without telling them how to do it, or why it’s crucial.

Think about all those semesters of Spanish classes in high school and college

Did they make you fluent? Of course not. How could something as technical as the statistical underpinnings of survey research possibly be any less difficult to learn? Mi tío tienes un lápiz azul.

Ratings, rankings, categories, and scales are loaded with misinformation traps undetectable by most

For example, when do you use odd-numbered or even-numbered scales? Did you know that people cannot reliably distinguish between more than five levels of agreement or importance? Did you know that most sophisticated statistical procedures require strict conditions that are typically ignored, making those analyses worthless? And on and on and on. Mark Twain’s observation applies here: “All you need in this life is ignorance and confidence, and then success is sure.”

You can’t judge a book by its cover

Neither can you judge the quality of the research by what you see in a report as slickly produced as Miranda Priestly’s Runway magazine. Be skeptical of the flashy presentation infographics that are there to distract you from the lack of meaningful content.

More surveys measure the wrong things than the right ones

A career middle manager in a multibillion dollar division of a highly-admired company had been assigned to repeat the same survey her boss had done every year. She told me they tracked several key indicators to determine how well their programs were performing. Each year, her boss included these numbers in his presentation to the president and they were subsequently featured in the annual report.

It was immediately apparent to me this survey in no way measured the things anyone thought it did

I told her that because new, accurate indicators would be entirely different, she would not be able to compare new numbers with numbers from previous years.

She had to decide

If she wanted to measure ABC, the current survey that measured XYZ was worthless and she would need a new one. If she used the same survey as last year it would be a pointless waste of resources.

She chose to repeat the useless survey

It would be trouble for her if she changed things now because her boss had written the survey years ago and was quite proud of it.

I declined the project

This incident is isolated only by the fact that the middle manager was honest enough to admit the situation called for CYA self-preservation over bringing bad news to her boss.

I cannot make an analytical silk purse out of a data sow’s ear

A potential client brought me a pile of data to analyze. He had written his own survey and collected his own data, then discovered he didn’t know what to do with it. His questionnaire had violated nearly every principle of good design and the data had been contaminated, so I told him I could not help.

Why I couldn’t help

Much like the chain of custody for medical tests and forensic evidence (who handled it, what did they do with it and to it?), surveys require several interlocking steps, and the failure of any one of them dooms the entire outcome. It is essential that the integrity of the data be maintained throughout the entire prescribed sequence. When it comes to collecting and handling data, all steps are critical. Start with a bad questionnaire like the one the potential client used, and the rest of the steps are meaningless wastes of energy, time and money.

Let’s say that somehow, against all odds, you have written a good questionnaire and have handled your data properly

You’re still stuck. Few people bother with the social sciences at universities unless they grab a Sociology 101 class as one of their electives. This is unfortunate because they didn’t get the training that would have allowed them to professionally analyze human behavior. The result is that the “analysts” in most businesses have the combined investigative prowess of Inspector Clouseau, Car 54, and the Keystone Kops.

Surveys are made unfriendly for users because they are designed backwards

Survey response categories and scales are all determined beforehand. The people being surveyed are required to shoehorn their thoughts into a fixed format. This is the same backwards engineering that brought us all those annoying telephone menus and all those damnable sites that send us to endless FAQ pages that answer every question but the one we have.

How many times have you filled out a survey that:

Went into too much detail?
Was too long and tiresome?
Made you rate and rank things that aren’t important to you?
Bored you so badly you answered mindlessly just to get it over with?
Yelled at you go back and answer those questions you left blank?

I don’t know about you, but when a survey does not allow me to skip a question, I skip the rest of the survey. I wonder what they do with my responses.

How to get the trustworthy customer, competitor and market facts you need

Get someone to take a look at your operation and tell you where you are doing well, where you need improvement, and where you need to be doing something entirely different.

How to avoid misinformation traps

Decision-makers don’t need to be research experts to upgrade their knowledge-management IQs. All they need to learn is:

How to tell good research from bad.
What questions to ask and how to assess the answers people give you.
How to scope research and allocate resources.
How to select, manage, and evaluate research vendors.
How to mandate process rigor and output quality.

Want to look at old things in new ways, see the commonplace in more detail and hear complex subject matter explained in simple terms?

Click here for the home page where you can scroll through the articles I’ve written every week for LetsTakeACloserLook.com since 2016, always showing there is more to most things than meets the eye. Scroll down to to the bottom of this page, enter your email address and get a new article in your inbox every Monday. There is no charge, I don’t share your address with anyone, I send you no e-mails, there are no ads on the site and I reply personally to everyone who sends me a note.