The Lost of Art of Pre-testing Questionnaires: Don’t Let Your Market Research Crash

I am stunned at how many experienced market researchers don’t bother pre-testing before they start data collection for survey projects.


It is the market research equivalent of a pilot who decides not to bother with the pre-flight checklist before takeoff.

I have had two recent experiences where I had seasoned researchers working with Research Rockstar clients, and they had assumed pre-tests were not required.  Really? That’s the assumption? I wonder how many pilots assume pre-flight checklists don’t apply to them.

There are certainly varying opinions about many market research best practices, but this really shouldn’t be one of them. Unless the survey research you are doing is a tracking study or an ongoing transactional study (in these cases the questionnaire has been tested, standardized, and assessed over time), pre-testing is critical.

Semantics: Pre-testing or Soft Launch?

I use the phrase “pre-test” and that is what I teach in Research Rockstar classes on project management and questionnaire design. Some people use the term “soft launch.” I am not hung up on the language, but there are some elements that are required in professional research regardless of your preferred lexicon:

  • Collecting responses from real research participants. A pre-test is not asking your Uncle Stan to take your survey and give you feedback. Sure, get Stan’s feedback—but before the pre-test, not in lieu of it. A real pre-test needs to be done with people from the actual sample source.
  • Using the final questionnaire. The pre-test must be done with the final instrument. Not a draft you know you will be editing anyway.
  • Using the intended data collection methodology. If it is an online survey, collect it online. “Phone” testing an online survey isn’t a true pre-test. Maybe it can be a pre-pre-test. For example, if you need to get feedback on answer options for a particularly jargon-full questionnaire, fine, do some phone work so you can find out how people are responding to answer options and wording. But that is not a pre-test.
  • Analyzing the results. It isn’t a pre-test if you don’t actually look at the results. There are several things we look for in a pre-test, but the most important one for many people is survey duration. This is a huge market research budget consideration—and can either hurt or help. So why not be precise? Especially for researchers who work with panel providers.  What if you told your panel provider the average duration would be 10 minutes, but your pre-test says 7? That’s real savings for you.

Pre-testing: Is Your Questionnaire Cleared for Take-off?  

For every 10 projects I pre-test, I may only make post-pre-test changes in three of them. Seven go forward, no changes needed. But the three that do get changes? Those are important. I have had pre-tests catch duration issues, programming logic errors, drop-out risks, and more. So yes, even though I have been doing this for 25 years, I still do pre-tests. Does it mean I don’t ever make questionnaire mistakes? Sure I do (in fact I had a doozey just recently, which I will post about soon). But pre-testing minimizes my risks.

Bottom line? Pilots have a re-flight check list that has 50 or more steps. We researchers don’t have quite that many on our pre-launch list, but pre-testing should be right at the top.


[Interested in more free market research tips and news? Want it delivered right to your inbox? Sign up for our newsletter here.]



Spring Training for Market Researchers

market research spring training

So many people have had a brutal winter—snow, snow and more snow! Makes you want to stay home, doesn’t it? With market research training classes in our virtual classroom, you can have fun, interactive education without the travel.

Check out these upcoming classes:

Introduction to Ethnography (single session, 1.5 PRC hours)
Tuesday, March 11, 2014 at 11am EDT

Social Media Meets Market Research (single session, 1.5 PRC hours)
Wednesday, March 12, 2014 at 11am EDT

Introduction to Conjoint Analysis (single session, 1.5 PRC hours)
Tuesday, March 18, 2014 at 11am EDT

Market Segmentation: Practical Steps to Research Success (single session)
Wednesday, March 19, 2014 at 11am EDT

Online Qualitative Research Methods (4-wk Power Program, 5 PRC hours)
Wednesday, March 19, 2014 at 1pm EDT

Managing Focus Groups (4-wk Power Program, 6 PRC hours)
Thursday, April 3, 2014 at 11am EDT

For our complete class catalog, visit Training.ResearchRockstar.com



Mobile Ethnography: The New ‘Organic’ Market Research Tool to Try in 2014

mobile ethnographyWhat’s the most promising aspect of mobility in market research? Mobile ethnographynot pushing surveys to mobile devices.

Mobile Ethnography: Innovation in Progress

While there are only a few tools available so far, this area is developing quickly.  Imagine being able to ask people to basically research themselves. They can opt-in to a research experience using their mobile phones, take pictures and videos of where they are, capture sound bites as they’re happening, scan barcodes or QR codes of interest, and so forth. Cool? Yes.

So what’s the downside? This market research technique isn’t perfectly controllable. Participants will vary in their adherence to instructions, volume of contributions, and time spent.  There will be inconsistencies, and surprises.

So like anything else, it’s a trade-off. Yes, there are inconsistencies—but for some research needs, mobile ethnography offers superior speed, respondent engagement and ultimately insights.  It’s not as structured as a “conventional” survey, but that’s ok.

Healthier Market Research?

I like organic produce. But it tends to be more inconsistent in appearance than “conventional” options. Similarly, some new ‘organic’ market research tools (like mobile ethnography), are a bit more inconsistent—but perhaps more nutritious. We researchers need to raise awareness with our clients, be they internal or external, that the flaws of some new methods are really cosmetic; that at the heart of new methods, we’re getting something that’s potentially a lot tastier.

Next steps?

Check out some of the early products. Three are below and, when you check them out, you will see they are very different from one another.

  • QualMeetings from 20/20 Research
  • EthOS from EthOS App, a UK-based firm
  • And the folks at MyServiceFellow are offering a free demo (as of January 2014—this may change at any time).


[Want to read more about organic market research options? Download our white paper here.]



3 of 17 Time Management Tips for Market Researchers

Time Management


A recurring challenge I hear from Research Rockstar students is that of time management. Too often, deadlines converge, fires erupt, or clients “need it yesterday.” So based on my 25 years of market research reality, I have put together 17 time management tips. You can download the full eBook here.

by Kathryn Korostoff


Create a “When Time Permits” (WTP) Folder

Do you ever get distracted by information that is interesting, but not urgent? If I’m not careful, I can waste an hour in an already-overbooked day doing something as trivial as reading reviews of some cool new software tool! A “catch all” WTP folder solves this problem by capturing those less urgent items that can still be important. It also reduces stress by preventing that nagging feeling that you may have forgotten something that is worth remembering. The folder becomes a simple, handy productivity enhancer during commutes or while waiting for meetings to start. Personally, my WTP folder is filled with sticky notes and scraps of paper where I have quickly jotted down products, people or book titles I want to look-up when I have more time. Rather than get distracted every time a new idea, topic or article comes to my attention, I can put it on hold until it makes sense.

Plan Ahead to Mitigate Known Risk Factors

All market research projects have risk factors. There are no exceptions. And most are known.

It is always wise to document your risk factors at the start of every project:

  • What are the top 3 risk factors for this project?
  • How can we mitigate them?

Risk factor example: unknown quality of list to be used for recruiting In-depth Interview (IDI) participants. Possible mitigation steps? Build-in time for a soft launch, identify a fallback list, and document risk factors for client (advising of specific benchmarks that need to be met).

By being proactive about risk factors, you will be able to act swiftly if a fallback strategy needs to be activated—and you will be able to set realistic expectations about time with your clients.

Avoid Analysis Paralysis

Market researchers often have piles of data to work with, and it can be hard to judge when the analysis is “done.” The most practical approach is to give yourself limits. Start by focusing on any analysis needed for the project’s documented objectives. If you are spending more than X hours (your call as to what “X” is for you) on analysis that is not directly related to your primary objectives, stop. For example, in one recent project, I gave myself 5 hours for some additional analysis that was really beyond the scope—but that I knew would thrill my client.

Yes, we often get tempted by additional analysis that can be useful to our client. But remember, you can always do more later, when the client isn’t waiting impatiently for your report. You can always meet your deadline first, and deliver an additional memo later showing your “above-and-beyond” analysis.

Want more tips? Click here to get your free copy of Kathryn Korostoff’s “17  Time Management Tips for Market Researchers” eBook.


[The Marketing Research Association (MRA) has approved many of Research Rockstar's live classes for PRC credits. Are you currently certified and planning for renewal? You now have 15 new options for meeting the MRA’s education requirements!]



Best Market Research Articles of 2013: Fourth in a Series of 10

Big data[Research Rockstar interns have written synopses of 2013’s best market research articles, as selected by Kathryn Korostoff. This is the fourth in our series. This synopsis was written by Research Rockstar intern, Audra Kohler.]

Article: Has big data made market research redundant?

Originally published in: research.

November 26, 2013

Brian Tarran

Big data.  The popular buzz word pops up constantly at market research events, in online discussions and in various industry publications.  In one 2013 article, “Has big data made market research redundant?”, author Brian Tarran summarizes The Debating Group’s event (a forum which asked four speakers to answer the article’s title question).

So did this event shed any real light? What is the role of conventional market research when big data is getting a lot of investment and attention?

Big Data and Market Research: Better Together?

Based on the responses of the event’s participants, big data seems to pick up where market research leaves off, and vice versa.  One speaker, Justin Sampson, chief executive of Barb, noted that big data fails to understand the context, and the social nature, of life.  Market research can fill that void.

Colin Strong, managing director of GfK Technology UK, stated that big data will “reinvigorate the market research industry,” and is yet another tool in the market research toolbox.

Big data and market research both provide equal value in information to the organization; at least according to Alex Chruszcz, head of insight and pricing at Asda.  As quoted in the article, big data gives “the what, when, how and who.”  The “why” is provided by market research.  However, Chruszcz believes that big data will soon be able to answer the ‘why’ soon (though we remain unclear as to how this will happen).

Based on Tarran’s article, it seems that the experts were trying to provide a balanced view: both types of data sources provide value.  But was it balanced because that is the true story, or because they were being politically correct? After all, it can be hard to speak to a room full of market researchers and tell them their role has become redundant because of big data.


[Don’t want to miss our next Best Market Research Article of 2013? Click here to subscribe to "Market Research Training from Research Rockstar" via RSS and have our upcoming articles delivered directly to your RSS feed.]