The Lost of Art of Pre-testing Questionnaires: Don’t Let Your Market Research Crash

I am stunned at how many experienced market researchers don’t bother pre-testing before they start data collection for survey projects.


It is the market research equivalent of a pilot who decides not to bother with the pre-flight checklist before takeoff.

I have had two recent experiences where I had seasoned researchers working with Research Rockstar clients, and they had assumed pre-tests were not required.  Really? That’s the assumption? I wonder how many pilots assume pre-flight checklists don’t apply to them.

There are certainly varying opinions about many market research best practices, but this really shouldn’t be one of them. Unless the survey research you are doing is a tracking study or an ongoing transactional study (in these cases the questionnaire has been tested, standardized, and assessed over time), pre-testing is critical.

Semantics: Pre-testing or Soft Launch?

I use the phrase “pre-test” and that is what I teach in Research Rockstar classes on project management and questionnaire design. Some people use the term “soft launch.” I am not hung up on the language, but there are some elements that are required in professional research regardless of your preferred lexicon:

  • Collecting responses from real research participants. A pre-test is not asking your Uncle Stan to take your survey and give you feedback. Sure, get Stan’s feedback—but before the pre-test, not in lieu of it. A real pre-test needs to be done with people from the actual sample source.
  • Using the final questionnaire. The pre-test must be done with the final instrument. Not a draft you know you will be editing anyway.
  • Using the intended data collection methodology. If it is an online survey, collect it online. “Phone” testing an online survey isn’t a true pre-test. Maybe it can be a pre-pre-test. For example, if you need to get feedback on answer options for a particularly jargon-full questionnaire, fine, do some phone work so you can find out how people are responding to answer options and wording. But that is not a pre-test.
  • Analyzing the results. It isn’t a pre-test if you don’t actually look at the results. There are several things we look for in a pre-test, but the most important one for many people is survey duration. This is a huge market research budget consideration—and can either hurt or help. So why not be precise? Especially for researchers who work with panel providers.  What if you told your panel provider the average duration would be 10 minutes, but your pre-test says 7? That’s real savings for you.

Pre-testing: Is Your Questionnaire Cleared for Take-off?  

For every 10 projects I pre-test, I may only make post-pre-test changes in three of them. Seven go forward, no changes needed. But the three that do get changes? Those are important. I have had pre-tests catch duration issues, programming logic errors, drop-out risks, and more. So yes, even though I have been doing this for 25 years, I still do pre-tests. Does it mean I don’t ever make questionnaire mistakes? Sure I do (in fact I had a doozey just recently, which I will post about soon). But pre-testing minimizes my risks.

Bottom line? Pilots have a re-flight check list that has 50 or more steps. We researchers don’t have quite that many on our pre-launch list, but pre-testing should be right at the top.


[Interested in more free market research tips and news? Want it delivered right to your inbox? Sign up for our newsletter here.]



Best Market Research Articles of 2013: Third in a Series of 10

[Research Rockstar interns have written synopses of 2013’s best market research articles, as selected by Kathryn Korostoff. This is the third in our series. This synopsis was written by Research Rockstar intern, Audra Kohler.]

Article: Are you thinking what I’m thinking?

Originally published in: research.

July 30, 2013

Rob Egerton and Jeanette Kaye

Have you ever bought something because all of your friends had it?  While we may be loath to admit it, our actions are swayed by friends, groups, and the public. Perhaps even more so than what we realize.  Because of this reality, the authors of “Are you thinking what I’m thinking?” argue that market researchers need to go beyond the individual to truly understand consumer behaviors.  The authors state that two particular theories should be used more in research to explore the dynamics of influence.

Wisdom of Crowds for Market Research

The author’s first cited theory, wisdom of crowds, was the theme of a popular 2004 book of the same title by James Surowiecki.  The basic premise is that group decision-making or estimation is more accurate than individual decision making.  An example: a group would be more accurate at estimating the number of candy corn in a jar at your annual Halloween get-together, rather than each individual guesstimating separately.  Another researcher, Martin Boon, took this conclusion one step further.

This is where the meat and potatoes lie in this article.  Boon reworks this theory to predict elections.  Based on his research with actual election results, he concludes that averaging a randomly selected sample’s guesses is more accurate than traditional polling methods.  The use of the wisdom of crowd’s theory had two clear distinctions:

  • Individuals were not asked how they were going to vote.  The sample was asked how they thought others would vote.
  • Previous election results were provided to each respondent, which provided a useful context.

Overall, this method proved to be more accurate than traditional polling.

The Theory of Group Behavior for Market Research

In his book “I’ll have What She’s Having,” Mark Earls makes the claim that in determining decisions, the influence of other people is more significant than the actual individual decision maker.  But if you think about it, as market researchers, we are great at knowing the individual and their thought process.  Rarely do we research how individuals behave in a group and how they are influenced by that group.

According to Egerton and Kaye, “…recent behaviors to which we can all relate point to how individuals can be encouraged into actions not by their own assessment of what they should do next, but by the actions of those around them.”  In Earls’ book, he cites the London riots of 2012, laying flowers at traffic accidents or at significant events as examples of group dynamics.

A Powerful Combination for Market Research

By integrating lessons from these two powerful theories, the authors create key market research lessons:

  • Acknowledge.  Realize that there are limitations to looking at only an individual’s behavior.  Behavior of the individual is influenced by group dynamics, the authors argue.
  • Explore.  Although this is difficult, the authors encourage beginning to map out how others influence an individual.
  • Categorize.  Egerton and Kaye cite a TED talk by Dereck Sivers, which gave a high importance to breaking down the behavior of early adopters versus followers. This is one way to start categorizing consumer behaviors by group.



New! PRC-certified Classes from Research Rockstar

PRC_FINAL_LOGO_minus_dropThe Marketing Research Association (MRA) has approved 15 Research Rockstar classes for PRC credits. If you are currently certified, or planning for renewal, you now have 15 new options for meeting the MRA’s education requirements!  Some of the classes now featuring PRC credits:

  1. 10-Point Checklist for Questionnaire Design
  2. Ask It Right: Choosing Scales & Answer Options for Online Surveys
  3. Intro to Ethnography
  4. Intro to Factor & Cluster Analysis
  5. Intro to Quantitative Data Analysis
  6. Market Research Project Management
  7. And many more!

We are thrilled that the MRA has provided this certification on such a wide variety of qualitative and quantitative class topics. For more details:

Sign up for classes today to get your PRC credits by end of year!

Any questions? Please contact Sales@ResearchRockstar.com.



One Big Survey or Three Small Surveys?

One Big Survey or Three Small Surveys 300x225When it comes to market research projects, how big is too big?

I see many clients struggling these days with a mismatch between the amount of market research data they have to work with, and the time they have to truly analyze and synthesize it.

Maybe it’s time to challenge assumptions about project size?  Instead of that next big survey project, could it actually be better done as three small ones?

Yes, I know the arguments for one mega survey:

  • One data set is easier to manage (clean, weight, etc.)
  • Same respondents answer all questions
  • Possibly lower data collection costs
  • Possibly lower incentive costs

But I also see lots of these situations:

  • Data getting stale while clients struggle to find time to truly dive in
  • Data being unused because it gets lost in the noise of “louder”—though not necessarily more actionable—findings
  • Customer lists that are being abused by monster questionnaires

What do you think? Might it be more effective to do a larger number of smaller projects? Could you challenge your plans for that next big survey? What would it really mean (pros and cons), if you were to break it into a few small surveys?


[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]



Best Market Research Articles of 2013: Second in a Series of 10

Rateocracy[Research Rockstar interns have written synopses of 2013’s best market research articles, as selected by Kathryn Korostoff. This is the second in our series. This synopsis was written by Research Rockstar intern Audra Kohler. Yes, technically this is a 2012 article, but we confess—we missed it until 2013!]


Rateocracy and its Impact on Market Research
RW Connect November 1, 2012
Robert Moran

Welcome to rateocracy, a world of public and nonstop rate-streams.  In this article, Robert Moran discusses the transition from traditional customer satisfaction as “proprietary and periodic” to the notion of rateocracy.

Think about eBay, Amazon, and Angie’s List—all public, continuous sources of ratings.  Moran cites a survey conducted by FTI Consulting which found that over 50% of respondents give a “great deal of consideration to online reviews from other consumers on sites…”  While such self-perception data is never perfect, it is safe to say that a high amount of people rely on ratings in order to help with purchase decisions.

The author describes three factors that will lead to an increase usage in rateocracy.  He describes an increase in the ratings culture, a “middleware” system, and the creation of an open and universal ratings system will drive society to rateocracy.

No matter how our society develops rateocracy, there are major implications for market research:

  • Rateocracy will speed the rate of the consumer feedback cycle.  This will happen both after an initial product launch and over the course of a product’s life cycle.
  • Quarterly tracking studies will become archaic.  On the other hand, the demand for analytics will skyrocket.
  • It will also increase demand on ad hoc research.  This is in part due to the increase in real-time information.
  • The focus will be on the “elite purchase influencers.”  This segment holds strong online reputations with the widest reach.  These market mavens are critical to any analytical study.
  • Platforms may change due to the increase in ratings.  Platforms such as eBay, Amazon and Angie’s List may offer a free level of analytics now, but continue to charge for advanced analytical tools.

Moran sums it up best, “rateocracy will accelerate the existing trends in market research toward tighter feedback loops, social media listening and big data.”

Buckle up.

[Interested in receiving free market research tips & news by email?
Click here to sign up for the Research Rockstar e-newsletter.]