Customer SurveysQuantitative Research

Open-ended Questions: The Home Exercise Machine of Survey Research

Americans happily buy $1,000+ pieces of home exercise equipment, with the best of intentions. But what happens when the slick new gear is unpacked and assembled? Nothing. Many home exercise machines are unused. Stationary bikes become expensive clothing racks. Elliptical become awkward home décor.

In survey research, we also waste an “expensive” item: the open-ended question.

Most online surveys rely heavily on “closed-ended” questions. For example, we often pose questions that are followed by five-point scaled responses (such as “very unlikely” up to “very likely”), or that allow the participant to “check one” or “check all that apply” from a list.

In contrast, an open-ended question is followed by a text box, in which the participant can type an answer. An example of this is, “Please describe your most recent dining experience at our restaurant,” or “What’s most important to you when purchasing frozen pizza?” How about: “For your next trip to a grocery store, what department will you visit first, and why?”

But sometimes we market researchers put open-ended questions in our questionnaires, and then neglect the results because analyzing them is too time-consuming. Despite all of the advances in marketing research software, the tools available for automating the coding and analysis of open ended questions are still underwhelming.  Indeed, even now in 2016, many experienced researchers use manual processes.

However, even just one or two carefully constructed open-ended questions can give you a wealth of information. It’s worth the pain, if they are well-designed, relevant questions.

Open-ended questions are especially useful for three types of market research goals:

  1. When you want to discover something: A great discovery question is, “What else can we do to improve your experience with our product?” or “What do you like best about X?” and “like least?” Simple questions like these can yield delightfully unexpected themes.
  2. When you want to hear people describe their opinion or attitude in their own words: What words do they use to describe their satisfaction or experience with something?
  3. When you want to measure awareness: Open-ended questions are a great way to measure awareness with the highest bar: unaided awareness. What is their top-of-mind awareness for a given product category? For an awareness question, do they name your brand? Do they name brands that you don’t think of as even being in your competitive set?

Yes, open-ended questions are “expensive.”  We are spending our participants’ time, and goodwill, by asking them to comply with these requests. We don’t want to annoy our respondents by asking for too much effort (yes, typing is effort), only to ignore their inputs.

Like the well-intentioned home exerciser, we want to avoid making the investment, only to waste it.

Yes, analyzing open-ends is hard work. So set realistic expectations: don’t put five open-ends in your questionnaire, if realistically you will only have time to analyze two.

Tags

Kathryn Korostoff

Kathryn Korostoff is founder and lead instructor at Research Rockstar. Over the past 25 years, she has personally directed more than 600 primary market research projects and published over 100 bylined articles in magazines. She is also a professor at Boston University, where she teaches grad students how to analyze and report quantitative data.

Related Articles

2 thoughts on “Open-ended Questions: The Home Exercise Machine of Survey Research”

  1. A fourth reason: 4. When you are fishing for compliments.

    With the popularity of NPS, many surveys will follow up on a positive NPS score and quote by asking permission to use that quote (anonymously or attributed) as a testimonial. Even those that don’t explicitly ask may mine such quotes to improve marketing messaging.

Leave a Reply

Your email address will not be published.