Let's Take A Closer Look

Explaining complicated subject matter simply since 1986

NINE STEPS IN THE RESEARCH PROCESS: An Overview of Using Critical Thinking and the Scientific Method to Gain a Competitive Advantage In the Marketplace

Good business decisions are based upon good information, and good information is the result of carefully planning and executing nine closely-related procedures, one at a time.

Studies begin when organizations determine they have a need for information to help solve problems or meet goals. The researcher meets with corporate decision-makers in confidential question and answer sessions. Together they…

Set the Business Objectives (Step 1)

The sponsors and stakeholders of any study must have a clear notion of what the goal of the research is to be. Executives with responsibilities for strategic planning, marketing, advertising, and other corporate activities work together with the researcher to determine organizational wants, needs, and goals. The researcher helps companies further refine their objectives by asking the following kinds of questions:

  • What do we hope to learn from this study?
  • How do we intend to use the information?
  • What level of accuracy will we require?
  • What resources do we have available to us to conduct this study?
  • Who will be responsible for each task?
  • How much time do we have to meet our goals?

When all involved have carefully examined these and other important issues, we know what we are trying to learn and we need to determine how we’re going to go about the process of learning, so we…

Establish the Research Objectives (Step 2)

Good research is neither mystery nor luck, but instead the result of careful and precise planning. Every good study needs a workplan that describes, in detail, what must be done to meet the business objectives of the study sponsor. The workplan specifies who is responsible for each stage of the study, when and where each activity will occur, and what will happen every step of the way. The workplan is essentially an outline of the research process. Key elements include specifying how to most effectively structure the learning experience and who we need to be learning from, so we…

Select the Research Methods (Step 3)

Research methods are the processes and procedures which objective information-gathering activities follow.  Research methods fall easily into one of two general categories: qualitative and quantitative. 

Qualitative research concerns itself with the nature of any set of human experiences, their essences and their characteristics. Quantitative research, on the other hand, is interested in making precise mathematical estimates of the rates of incidence of those sets of human experiences. Simply stated, qualitative methods seek to find out how, while the goal of quantitative procedures is to find out how many. 

The best, most thorough research capitalizes on the strengths of both methods

The logical sequence is to begin with conducting individual and group interviews to establish the range and types of behavior occurring in the population under study. Seen by some as little more than rooms of people chatting aimlessly, the best focus groups are in fact thoroughly planned, carefully organized interactive discussion sessions. Because this method of research is exploratory in nature, findings are treated not as conclusions, but as testable hypotheses, becoming the foundation for the stricter, more rigorous discipline of survey research.

Survey research is structured and formal, following strict mathematical principles to arrive at conclusions that can be generalized to populations at large, and subsegments of those populations.

While studies involving only a single method are generally considered to be narrower in scope and to have more limited applications, they are quite appropriate for meeting certain specific objectives.

At any rate, now that we know what we want to learn and how we’re going to go about learning, we need to decide who we are going to learn from, so we…

Choose the Sample (Step 4)

When we collect information from everyone, from an entire population, we are conducting a census. In the United States, we conduct our national population census only once every ten years because it costs so much, takes so much time, and involves so much work.

If the results of a particular research project have extremely important implications, or if the size of the population under study (the universe) is very small, then a census is the appropriate way to proceed. But generally, if the group we are studying includes more than a few hundred people, or is widely dispersed geographically, we want to use a sample, or a subset of the universe.

Good samples must meet very exacting criteria for validity, reliability and representativeness. Many non-researchers find technical terms (industry jargon) like this somewhat confusing, but most agree the underlying concepts are easily understood:

  • Validity is a measurable quantity that expresses the soundness of the work.
  • Reliability refers to the extent to which a piece of work is dependably accurate
  • Representative work allows us to draw conclusions and make generalizations about our results that apply to the population at large.

Determining the sample size is another important activity because sample sizes are subject to precise, exacting mathematical standards. They also must be large enough to deliver analysis at the level of detail specified by the business objectives outlined in the workplan.

As data collection is a sizable proportion of the cost of conducting consumer studies, sponsors and researchers must select a sample that achieves a balance between precision and expediency. Generally, the more detail required and the broader the scope of the work, the larger the sample must be to give us adequate confidence in the accuracy of our findings. Every research study has a margin for error, and the good ones have determined what the necessary level of confidence is in advance.

The nature of the sample is every bit as important as its character and its size. One often-used type is a quota sample, where subjects are selected because they have very specific qualities that are of particular interest to sponsors and researchers. Another basic type is the random sample, where each member of the universe has an equal opportunity of being selected for study.

Almost no one today uses a true random sample

One company executive objected when I said this, saying “We do.” He did not understand his research was choosing targeted subjects at random from a pool that is not random. Random costs too much and takes too long, arguments that appeal to cheapskates and the irrationally impatient. Online samples are used by many companies as a universe when they aren’t. They are pools of pre-recruited people who have agreed to participate in research studies for compensation. This, of course, leaves out all the rest who weren’t pre-recruited and those who were recruited but weren’t interested. Before your organization hires a research company that uses panels for its samples, ask them what steps they take to weed out the cheaters, repeaters, survey pros, frauds, speed demons, and those with no interest in following the rules. It’s easy to understand why they shouldn’t be in the pool. By the way, is your sample further compromised by being restricted to Facebook users, for example?

While collecting information is easy, collecting good information that is accurate, precise, thorough, and delivers on our business objectives requires a great deal of planning, work and attention to detail.

Once we’ve found the people we need to learn from, we had better be certain that we are asking them the right questions, the right way, and in the proper sequence. With this in mind, we….

Design the Testing Instruments (Step 5)

As is the case with methods and samples, researchers have a variety of choices at their disposal when it comes to testing instruments. Researchers design topic outlines and discussion agendas for personal interviews and questionnaires for surveys. Each must be carefully planned to extract meaningful information from study subjects.

Like all good written work, testing instruments go through a series of drafts and revisions as they grow to meet the study’s business and research objectives, which typically continue to evolve over the course of planning and designing the research methods, samples and testing instruments. When this happens gradually, it is called scope creep.

Focus group discussion agendas and interviewer topic guides need to be flexible to allow for the ebb and flow of true interactive conversation. Surveys, on the other hand, must be more rigid and precise because they are to be applied to each subject in the same careful manner, over and over again, be they face-to-face, online, or older techniques still used outside the U.S. by telephone, or by postal mail.

Once all these issues have been raised, discussed, and settled, we need to get on with the activity that brings us in direct contact with the study subjects, so we…

Collect the Data (Step 6)

Carefully-kept records provide the building blocks of the objective, rational, and analytical model. For individual and group interviews, researchers collect session data by recording the entire proceedings. On the quantitative side, personal interviews typically yield better data than self-administered questionnaires because trained interviewers can use interactive procedures such as probing techniques with respondents to clarify answers and provide richer responses. 

When we’re finished with the task of collecting the information, we must do something with all of it, so we…

Process the Data (Step 7)

Data processing clerks might work with audio and video session tapes to make verbatim transcripts of everything said by the moderator and the participants throughout the course of the discussion. Bigger research companies use machines to do the work for them.

Surveys require far more complicated handling and much stricter supervision. First, all surveys must be edited for accuracy and completeness. Next, specific numerical codes are assigned to every response and all survey items are entered into a single, custom-built database. 

When all the data are loaded into computers, programmers set parameters and write sets of statistical procedures that read the database and sift and sort the information in every way imaginable. In the final data processing procedure, analysts construct tables to display the information in clear, precise terms that compare and contrast findings. Here the researcher begins what I consider to be the most challenging and rewarding step of all; here the researcher has the opportunity to…

Analyze the Results (Step 8)

Analysis begins when the researcher creates a conceptual framework, a theoretical structure within which to review the findings, and follows a strict analytical methodology.

Good researchers know the results of any course of inquiry are within the information itself and can be discovered only by proceeding carefully, cautiously, and deliberately. They trust the scientific method and draw upon their understanding of human attitudes, values, wants, needs, and behaviors when analyzing quantitative data.

A good researcher works to solve these intellectual puzzles by exhaustively reviewing transcripts and tables with the following thoughts firmly in mind:

  • Be objective.
  • Be disciplined and precise.
  • Be intellectually curious.
  • Be constructively critical.
  • Be thorough and complete.

When analyzing interview and focus group findings, the researcher carefully sorts through the data to determine which comments have value, what themes lie beneath the surface of the comments, and how everything fits together. Again, high-end research providers have automated this process. The procedure is much the same for reviewing survey results, but with one large difference: the researcher can ask statisticians to conduct more finely detailed and demanding procedures to look ever more closely at the data.

The final stage in this extremely rigorous process is to effectively communicate the results of all this work, so we…

Present the Findings (Step 9)

When writing about a study, good researchers communicate the findings, their deductions and their inferences simply and clearly. A good report not only lists the results, but also draws concrete conclusions and makes actionable recommendations. Beware reports that are top-heavy with jargon and unnecessary mathematical detail.

As with the planning and testing phases, top-notch research reports must go through several drafts before the work can evolve to become meaningful documents that sponsors can use to confidently improve current products and services, design new ones, and find new consumer markets.

Most study sponsors would rather look at fancy slideshows than go through a written report. Good researchers know more than statisticians about the graphic arts and more about the graphic arts than statisticians. For more, visit EdwardTufte.com.

The remarkable beauty of the scientific method is this:

When rigorously followed, a good researcher can use it to study any problem with the skilled detachment needed to really understand an issue with minimal bias and limited assumptions, and to objectively pursue learning what really exists among the hearts and minds of the people who know best: the consumers.

Bonus

Ron Sellers, writing at Greenbook.org, says his team sent a typical online questionnaire to five of the ten largest panel providers in the research industry.

Grey Matter Research and Harmon Research set up tests, traps, and quality control measures to learn the extent to which problems exist among online panels. Ron says they tossed nearly half (46%) of the respondents out of the study for having serious and egregious problems. Random sample, indeed.

Having split off the nearly 900 junk panel members they removed from the study, the researchers ran the numbers so the could compare junk responses with those from valid respondents. Here are three examples of what happens to your data and your findings when your sample has been compromised. 

  • When asked about their awareness of a named charity watchdog organization, fifteen percent of the valid study subjects claimed they were aware of the organization. Compare their 15% with the 58% of fraudulent subjects who made the same claim. Why do you think that huge gap exists?
  • Brand familiarity is one step higher than brand awareness and one step lower than top-of-mind awareness. Three percent of valid subjects claimed they were familiar with two fake banks with made-up names. Ten times as many bogus subjects claimed to be familiar with banks that don’t exist. When asked about their familiarity with three real banks, bogus subjects claimed rates 44% higher than did valid subjects on the first bank, 167% higher on the second, and 400% higher for the third. Even non-statisticians can tell those numbers are due to something other than chance.
  • Asked to carefully read a 200-word concept statement before answering some questions, valid subjects took an average of 80 seconds reading 200 words. bogus subjects averaged 11 seconds. As Sellers points out, that difference would wreck any concept test.

Their experiences working with tens of thousands of online panel study subjects led the researchers to conclude that “at least 90% of researchers are not taking sufficient steps to ensure to panel quality.” Random samples, indeed.

You can read about these issues and findings in Dirty Little Secrets of Online Panels and Still More Dirty Secrets of Online Panels.

 

Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Loading