Best PracticesCustomer SurveysMarket Research

Survey Design 2011: NEW REALITY, NEW RULES

Obsolete_Computer_In_The_TrashLet’s get a few facts on the table, shall we?

  1. We all have non-researcher friends, colleagues and acquaintances who are writing their own online surveys using one of the many free or low-cost tools.
  2. Despite all of the well-publicized voices shouting, “The survey is dead!”, “DIY” survey volume continues to surge.
  3. We may tire of hearing, “The survey is dead!”, but we all know that the role of the survey isn’t what it used to be, and we really aren’t that sad about it. Indeed, many of us now spend more time on non-survey methods, either to develop best practices or for “live” projects.
  4. Short surveys are better surveys.
  5. New tools and technologies create possibilities for survey design and data analysis that never existed before.

So, what does this hodgepodge of facts mean?

The role of the survey is changing. Who creates them, how they are designed, when they are used, and their role relative to other methods is all shifting, fast. To remain effective in this new reality, survey research best practices must change, and the time is now.

New Rules of Survey Design

Some of the New Rules are driven by length, others by what is feasible given recent advances.

Then: Three to five screening questions were acceptable.

Now : One or two. Better yet, none—just collect the profiling data from your MROC database or panel provider.

Then: 15 to 20 minutes average duration was the old “short” compared to the 40 to 60 minute monsters of the old days.

Now: Ten minutes or less.

Then: Multiple questions per screen to minimize number of total screens.

Now: Shorter surveys make it worthwhile to exploit the visual appeal of one question per screen.

Then: Two to three open-ended questions at most, due to both respondent compliance and analysis headaches.

Now: Wildcard. We still want to avoid demanding too much “work” from our respondents, but these questions have become much easier to analyze. Whenever an open-ended question could replace a question with a long list of answer options, it’s worth a look.

Then: Survey design and programming were done by a professional researcher, going through multiple iterations, often taking days if not weeks.

Now: Basic survey design can be easily done by any motivated professional with the help of training, survey templates, or other support.

Then: Surveys designed with the assumption that they are the primary data source for an overall project.

Now: Surveys designed with the context that the survey is only one source of data, and is likely to be used in combination with multiple data sources.

Then: Surveys designed with the assumption that there would be one mode of data collection (typically phone, online or paper).

Now: Surveys designed for multiple modes of data collection, possibly including mobile.


I am generalizing here, and there are plenty of “buts” and “exceptions.” The reality, however, is that the landscape of survey design has changed. We can’t let yesterday’s best practices hold us back today.

Bottom Line

Those of us who remember spending hours (even days) to craft what can only politely be described as visually bland surveys can be forgiven a secret twinge of envy when we see DIYers creating visually stunning surveys in under an hour. But we have to get over ourselves; new tools are in, and the old rules are out.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]


Kathryn Korostoff

Kathryn Korostoff is founder and lead instructor at Research Rockstar. Over the past 25 years, she has personally directed more than 600 primary market research projects and published over 100 bylined articles in magazines. She is also a professor at Boston University, where she teaches grad students how to analyze and report quantitative data.

Related Articles

4 thoughts on “Survey Design 2011: NEW REALITY, NEW RULES”

  1. Kathryn – passionate piece, great read. Beg to differ on some of your statements – especially this one: “We can’t let yesterday’s best practices hold us back today”: why should best practices hold us back? I think it’s essential to learn from decades of research experience. Yes, a lot has changed, and a lot is changing – but i think we need to encourage good practice in research and survey design and not simply throw the baby out with the bathwater. I think that path leads to an erosion of credibility for us as a profession, and there are plenty of poorly informed non MR-folk out there who think they’re writing a great questionnaire when in fact they could be writing an even better one, with lots less bias for example.

  2. Thanks Edward. My take is that some best pratcices are now outdated–though certainly not all of them.

  3. Kathryn:

    The landscape certainly has changed and we need to evolve with the changing landscape. I would not encourage abandoning yesterday’s best practices without seeing how they can be retrofitted to fit today’s realities.

    As always, test new methods and see how they compare to the norms before throwing out yesterday’s hard work.

    Thanks as always for raising the collective consciousness.


Leave a Reply

Your email address will not be published.