How Did Businesses Waste $2 Billion on Focus Groups?
Two slick-talking tailors promised to make a suit of clothes that was so special it would be completely invisible to stupid and incompetent people. The emperor ordered a special suit and later dispatched his most trusted ministers to see how it was coming along. The con men, working away at their empty looms, explained the fabric they were weaving in extravagant detail, declaring it Captivating! Breathtaking! Enchanting! The ministers, fearful of being exposed as idiots, went back to their sire and told him the fabric, the colors, and the design were splendid and glorious.
When his new suit was delivered, the emperor was stunned that he could not see anything
It would never do for the emperor to be stupid and incompetent, so he gave the suit his highest praise. None in his entourage could see a suit, either, but all joined the emperor in praising its wonderfulness. After having him undress, the swindlers elaborately dressed him in his new “clothes.” After much ooing and aahing by the con men and the courtiers, he took his place in a great procession through the town so everyone could admire his finery. None of the people watching the parade wanted to appear stupid and incompetent, either, so each praised the new clothes more loudly than the next – Magnificent! Majestic! Spectacular! – until a small child shouted, “But he doesn’t have anything on!”
Hans Christian Andersen wrote this satirical tale to expose hypocrisy and snobbery
It is popular with children because the child is the only one who has the courage to speak the truth.
The right tool for the right job
That was one of the first lessons I learned when I was a mechanic, and from what I’ve seen, very few in white collar jobs know how important it is. Businesses throw away billions of dollars every year because they use clocks to measure the size and weight of chairs and screwdrivers to pound nails. Tradespeople have known for centuries how important it is to use the right tool for the right job. Too bad the people who spend billions on focus groups don’t know that simple truth.
Businesspeople conduct focus groups for some very bad reasons
- Most want to be able to report their research proves the boss’ pet project has full and enthusiastic consumer support. Focus groups are ideal for this because participants can be easily steered to provide any results sponsors desire. Complicit vendors use cooperative moderators who stack the deck with leading questions, praise all positive responses while ignoring all the negative ones, and put the spotlight on those who say what the client wants to hear.
- Most treat focus groups as if they were surveys (mathematical data). Nothing is more absurd than moderators asking for a show of hands and then counting these responses as if they were part of a statistically valid sample of randomly selected participants. If your moderator does this, get a different one.
- A surprising number of sponsors enjoy the voyeuristic feel of eavesdropping on others while hiding in the dark behind a mirror. Erika Hall of Mule Design calls focus groups “useless research theatre.” In her article, Focus Groups Are Worthless, she says “No one buys shoes or cooks dinner sitting at a table under fluorescent lights while engaged in a moderated group discussion.”
- They don’t know that focus groups are for discovering questions, not answers.
Recruiting
Many vendors stack the deck in your favor. They want their clients to be happy and know that comes only when the research supports what the clients want to hear. If you’re bringing out a new line of healthy foods, they’ll recruit people who eat only healthy foods. This naturally pre-ordains the outcomes so clients can show their bosses how study subjects are gaga for all their new products.
Social desirability is always an issue
In groups, people tend to say things they think will be viewed favorably by others. This includes under-reporting “bad” things and over-reporting “good” things. Here’s one of my favorite illustrations of how focus group conclusions can be so out of whack with the scientific evidence.
- People in focus groups claim to have healthy eating habits.
- Companies sell billions of dollars in unhealthy foods every year.
- Two-thirds of Americans are overweight.
- Obesity-related conditions include heart disease, stroke, and Type 2 diabetes.
Most of us learned the hard way that what people say isn’t always what they do
Nowhere is it more evident than in focus groups, where people are paid to go to a strange place so a stranger can ask them to reveal their innermost feelings to a roomful of strangers while being observed by still more strangers.
The entire research industry knows how what people say and do are seldom the same thing
This is why userfocus.co.uk asks “Why does your company expect focus group participants to:
- Inspect their cognitive machinery?
- Understand what they find there?
- Translate those thoughts into spoken words?
- Articulate their thoughts unambiguously?
Not every group member is a well-informed, insightful, and outspoken person
The more corners vendors can cut, the more goes to profit. The result is that too many of the people in your focus group are the wrong people to be asking about your products and services.
Not every focus group participant is an amateur, either
Less-scrupulous vendors employ ringers, people who lie about their experiences. These are people who research firms hire over and over again because they can be counted on to be positive, persuasive, and follow instructions. If these stooges were boxers, they’d be the pugs willing to take a dive. What most decision-makers don’t know is the positive things ringers say about their products and services are untrue.
Less-outgoing participants tend to agree with outspoken and assertive alpha participants so paid ringers steer the group in predetermined directions with or without the help of the moderator.
Validation is for parking lots
Many companies that pay for focus groups don’t want to hear anything negative. They want to hear participants tell them how wonderful their products and services are. They are seeking validation and lots of research companies compete with each other to get their business.
Terry Linhart of Arbor Research Group often politely asks clients if they really want to know the facts or if they just want the research to confirm the genius of what they’re already doing
When I met clients who clearly wanted confirmation and not exploration, I would tell them we could find out exactly what they wanted for a large fee, or for half that amount, we could find out what actually exists.
I am not saying don’t use focus groups — what I am saying is don’t use focus groups for the wrong reasons
Robert Merton, the inventor of the focus group, despised how his methodology had been hijacked and bastardized, saying “Focus groups are supposed to be merely the source of ideas that need to be researched.”
Focus groups were meant to be only a jumping-off point, with perhaps a hypothesis or two thrown in
Instead, focus groups became a favorite way to get quick answers by taking fatal shortcuts. Listen to what some other folks have to say about focus groups:
- Advertising genius David Ogilvy summarized the trouble with focus group research is that “people don’t think what they feel, they don’t say what they think, and they don’t do what they say.”
- Daniel Gross, author of Lies, Damn Lies, and Focus Groups, says “The widely documented mismatch between what people say and what they do costs companies millions.”
- Kay Polit, principal analyst at management consultant company A.T. Kearney, says focus groups are “a faulty process.”
- Mary Quinlan says they are “a dangerous way to get market intelligence.”
- Joseph Pine and James Gilmore, authors of The Experience Economy, call focus groups “the great lie.” They say focus groups should be dumped and real customers observed in real settings instead.
- Dev Patnaik calls focus groups “a customer terrarium,” with people taken out of their natural surroundings and put behind glass.
Focus groups do not tap into participants’ true feelings, no matter what someone may have told you
People have complex and conflicting motivations. These are impossible to get at with the ridiculously-detailed discussion guides so many moderators rush through in a vain attempt to meet stakeholders’ conflicting demands. Focus groups also ask participants to make snap judgments about products they’ve never seen or used.
In his book How Customers Think, Gerald Zaltman says 80% of new products fail within six months of being vetted in focus groups
Why the lousy results? According to Zaltman, “The correlation between stated intent and actual intent is usually low and negative.”
If your gatekeepers and research providers haven’t told you these essential truths about focus groups …
… they’re either withholding crucial information from you — which is bad, or they’re unfamiliar with the facts — which is also bad. You lose both ways.
No matter which, you should immediately seek new gatekeepers and vendors because they’re selling you one magnificent, majestic, spectacular suit of clothes after another.
Want to screw up your focus groups? Try these proven techniques:
- Overbuild your discussion guide. Instead of identifying some broad areas to explore thoroughly, go ahead and produce a rat’s nest of unconnected issues so long that study subjects can’t possibly give tcareful consideration to any of them.
- Use complicit moderators who are skilled at transmogrifying roundtable discussions into camouflaged sales sessions that flatter your business.
- Produce faux quant. Ask participants to vote on things. Tally responses so you can later claim that 75% love your idea.
- Avoid giving participants the opportunity to think carefully. Pressure them to provide answers that fit your expectations. Keep on pounding at what you want to hear. Listen to only the praise and ignore the criticisms.
- Send in notes. This not only interrupts the flow of the discussion, but also reminds participants that they are being watched.
- And by all means, use them to validate your already-made decisions instead of asking questions.
To make absolutely clear to our clients what focus groups are good for, we developed this definition:
“Focus groups are one type of exploratory research that we use to guide our survey research.”
Focus groups are best for:
- Learning what regular people really want to talk about instead of what industry experts want to talk about.
- Uncovering regular people’s real questions and issues.
- Learning the words consumers use to describe our company and our products.
- Identifying not only commonality, but variance, too.
- Writing better questionnaires.
Smarter businesses have moved on to better ways of conducting qualitative research, replacing focus groups with ethnographies, on-site observations, one-on-one and small group interviews, and a host of hybrid methodologies.
How to get the trustworthy customer, competitor and market facts you need
Find someone to tell you where your research is top-notch, where you need to fine-tune, and where you need to be doing something entirely different.
How to avoid misinformation traps
Decision-makers don’t need to be research experts to upgrade their knowledge-management IQ. All they need to learn are these five things:
- How to tell good research from bad.
- What questions to ask and how to assess the answers people give you.
- How to scope research and allocate resources.
- How to select, manage, and evaluate research vendors.
- How to mandate process rigor and output quality.
Want to look at old things in new ways, see the commonplace in more detail and hear complex subject matter explained in simple terms?
Click here for the home page where you can scroll through the articles I’ve written every week for LetsTakeACloserLook.com since 2016, always showing there is more to most things than meets the eye.
