There are a lot of cases where what you are trying to capture, whether it is customer attitudes or behaviors, would be best served using a different type of scale or answer option.  For anyone who designs online surveys and is interested in making sure that they are collecting data that is both objective and complete, a good hard look at how you choose your answer options and scales is critical.

In our new class, “Ask It Right: Choosing Answer Types & Scales”, Research Rockstar covers Likert Scales, Semantic Differential Scales, Constant Sum, Ranking Questions, Rating Questions and more.  We specifically highlight examples of how to structure answer options for common research market needs, including customer satisfaction, product concept testing and more.

The class also includes several examples of “bad” questions, and how they can be rewritten to be better.

In market research, we all know we need to make sure that we are collecting data that is going to give us a complete, objective view of a target population’s, attitudes and behaviors. As important as it is to word the questions carefully, it’s equally important to make sure answer options are going to make sense.

Research Rockstar VIP, Backstage and Tour Bus members all have access to the class now; please just visit your member’s pages. Not currently a VIP, Backstage Pass or Tour Bus member?  The class is available for individual purchase here.

We particularly recommend this class to anybody who has colleagues who are new to questionnaire design. This is a very practical class that will help them avoid the common mistakes made by novice market researchers.

I’m reminded of raising a teenager, who reported that she “just had to have” a certain pair of shoes. They were stylish. They were unbelievably comfortable.  And all her friends were getting them. In good parental fashion, I struck a deal. We’d wait a month until report cards, and if her grades met expectations, we’d consider the purchase of $100 shoes.

A month later the grades were fine, but the shoes were no longer desired. Turns out nobody was wearing them anymore because they were too tight, fell apart and looked funny. Yet had we gone with her initial “analysis”, we’d likely have concluded that this was a great product — and we’d have been wrong. So how do we avoid the trap?

Market Research & Insight

As researchers we have to keep ourselves honest.  A few coincidences do not a key finding make, and I’ve seen some frightening cases of “conclusions” based on weak data analysis and flimsy proof points. This isn’t value—it’s hyperbole.

During analysis we can’t assume that “interesting patterns” reflect a broader reality, nor force a conclusion where none is justified.

Here are two best practices to help practice the discipline of meaningful insight creation:

Challenge your own conclusions

If you’re doing a research study and the data seems to perfectly support all your hypotheses, you need to ask yourself whether your research or the process was biased.  Perhaps your research was intended to confirm things your organization already knew. But if you’re hoping for some real “aha” points, this may not be what you want.  If your research results wrap up too neatly, look closer.  Is there an opportunity to challenge any of these results?  I’ve done several customer loyalty studies that at face value gave very positive results—customers were broadly satisfied and loyal. Digging deeper, though, we learned that important segments were loyal, but due to existing contracts and cost-of-switching issues — externals that dramatically changed the story.

One way to challenge your conclusions is to seek contradictions.  Look at how specific key attitudes and patterns vary by subgroup.  Examine differences by gender, age and geography, or if it’s a B2B study, by job function, industry and company size. Does the story still hold?  Sometimes the obvious is masking something that might require a bit more intellectual rigor.  As researchers, it’s our job to be disciplined and get beyond face value.  What’s the story behind the story?

Validate your results

We have to validate our data, especially when the research conclusions seem too-good-to-be-true. Whenever possible, validate by comparing with other data sources. You want to determine if relevant external data points also show consistencies about the customer attitudes and behaviors you’re examining in this particular study. A couple of examples:

• In a recent survey project, the client used social media monitoring to find customer stories and anecdotes that reflected behaviors measured in the survey project. Not the most rigorous methodology, perhaps, but it did provide real-world examples.
• In many studies, I have had clients doing online surveys conduct a small number of follow-up IDIs with respondents to access some of the “why” context.

Market Trend or Coincidence?

Sometimes as researchers we get so excited about our data that we jump.  But remember, a few coincidences don’t justify a conclusion.  We have to make sure that the story we’re sharing can be confirmed, replicated, and demonstrated with solid proof points. Otherwise we must present those conclusions as hypotheses, or “directional”. That’s not necessarily a bad thing, but it is different, and we have to have the discipline to recognize the difference. Otherwise we’re just peddling bad shoes.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Do you question whether your choice of scales and answer options is yielding the best data?

Has your organization ever released an online survey—only to get results that are too vague to be actionable?

Let’s be honest—a lot of people with no training write surveys these days.  And it shows.  If you or your colleagues are in New York please join me for a crash course in online survey design.

All students will receive a workbook and online access to additional materials.

Our agenda:

• 5 steps to planning a successful online survey
• How to avoid common survey design problems
• All about question types and scales
• How to minimize bias and maximize honesty
• Tips for getting people to complete your questionnaire

This event is being hosted by Research Rockstar, the NY AMA, and Greenbook.

Learn more | Register today

How does your target market perceive your brand as compared to your competitors’ brands?

While brand research can be a very complex, exhaustive exercise, in many cases a simple approach may suffice. 

If you plan to do your own brand perception research using online surveys, here are some tips.

How your brand is perceived

For brand perceptions, a quick and easy way to collect data is to ask, “Which of the following words would you use to describe our company?” Then give them a list of varying words and allow them to pick up to three. It’s a simple format for the respondents, and gives you very useful insights.  Do people think of your brand as “smart” and “fun” or “stable” and “safe”? Are your competitors perceived as “friendly” and “creative” or “slow” and “boring”?

Other perceptions that we commonly seek to measure in research:

• This is a company that values its customers
• This is an innovative company
• This is a company that offers products or services that are a good value (or a good value for the dollar)

These types of brand questions are going to vary by product category and target market. B2B companies will have very different questions than B2C, and so on.

Brand Perception Research, Realistically

In an ideal world, a company would do very comprehensive brand perception research. But that type of time, and budget, is not always an option. With some careful planning, many companies can learn quite a lot from a short, online survey approach.

If you’d like to receive more free Market Research tips, click HERE to sign up for Research Rockstar’s Market Research Newsletter.

Without the normal crush of deadlines, market researchers can create templates with these goals in mind:

1. Set standards.  Use this time to think carefully about how you want to standardize specific question types, along with formatting and scaling options.
2. Get approvals. Get approval from those colleagues or managers who will be involved in year-round research efforts. Explain you are constructing standard templates and want their input during the slow season—this will help everyone come crunch time.

For those newer to research, you will find that there are three common survey templates that come in handy.  A few examples of question types are included below, but there are many options—so be sure to look at several examples before you craft your templates.

Survey Template #1: Customer Satisfaction Research

A simple satisfaction survey would consist of four or five questions to gauge satisfaction and loyalty. Of course, the type of customer satisfaction and loyalty questions depends on whether you’re selling business-to-business or business-to-consumer, whether you’re selling services or products and so forth. In general, you’re going to want an overall gauge of satisfaction, which commonly uses a five-or seven-point Likert scale question. Add a few follow-up questions about satisfaction relating to the specific aspects of your products or services, as relevant, such as customer service, product’s ease of use, and possibly aesthetics.

As an example of how this might need to be modified for different product categories, let’s consider a snack product company. For this case, satisfaction measures might focus on the variety of flavors offered, response to specific flavors, and package size.

Survey Template #2: Website Feedback

If your organization interacts with customers on its website, it’s good to have a standard template for collecting website feedback. This could be used on either a transaction or a rotation basis (so that customers see it on every 10 or 20 visits), or maybe it’s something you will use once a quarter.

Common questions collect feedback on overall attractiveness, distinctiveness, and ease of use. So answer options might use a scale of “very mundane” to “very exciting”; or maybe a range from “amateurish” to “very professional”; or perhaps, “not at all easy to use” to “very easy to use”.

Other key questions may include:

• “Were you able to find the information you were looking for on our website?”
• “How likely are you to visit this web site again in the next 30 days?” (or whatever timeframe would make sense for your particular category).

Survey Template #3: Customer Service Transaction

You may want to have a survey that’s triggered every time someone completes a support call or other type of customer service transaction with your organization. This could be done through a call center, email, or even through a social media interaction such as on a Facebook fan page or via Twitter.

Typical questions ask about:

• How quickly they got a response, which gives you an objective assessment of whether or not it was timely
• Their satisfaction with the timeliness of the response
• Their satisfaction with the quality of the response

Moreover, also use it as an opportunity to make sure that the matter was completed successfully so that you can create a red flag if necessary

Market Research Planning

In my town, the Department of Public Works knows that we’re going to get a lot of snow each winter, and you can be sure that for the couple of months before winter comes, they’re stockpiling salt and sand for the roads. They’re using that relatively slow period before crunch time hits to make sure they’re prepared.

Well, it’s the same thing for survey writing. Let’s take advantage of this slow time to make sure that we have everything we’re going to need before the next crunch time hits — as it inevitably does.

Click here to check out Research Rockstar’s full line of Online Market Research Training Classes.

1.   Too many open-ended questions.  Very often, when we’re writing a questionnaire, we realize that there are opportunities to discover new things or we are unsure of which answer options to offer. Our solution? Open-ended questions. We might ask people about unaided brand awareness; that is, “When you think of Product Category X, what brands come to mind?”  Or you might ask, “What else can our company do to improve your satisfaction with our services or products?”  Or you might have a question followed by a list of possible answers, including an “Other. Please specify:                      ” option. The first two examples are open-ended questions, but even the third one is expecting a lot.  Having two or three questions that require real writing is fine.  But if you ask too many, it becomes a turn-off. It is simply too onerous and few respondents will type that much.  The result? You end up with a lot of missing data.  So choose wisely, and use open-ended questions judiciously. 

2.   Excessive jargon.  If you’re doing a survey project, chances are that you have a great deal of expertise in a particular product category, industry, or topic area. And by virtue of the fact that you’re an expert, you have developed a specific language for talking about relevant issues. It’s very easy for those of us who develop areas of expertise to forget that other people simply don’t use the same language to discuss the same topics.  We have to be vigilant when we’re creating surveys to use friendly language.  Go for the lowest common denominator in terms of who’s going to be taking your survey—and use language that they are likely to use.  Excessive jargon turns people off and leads to dropouts, or worse. If they don’t really know what a term means they might guess, and you might be getting inaccurate data in return. 

3.   Forgetting your manners. It sounds trite but it is really true.  We need to be respectful of the people who are taking our surveys.  An occasional “please” and “thank you” goes a long way.  In the survey opening, use polite text to set the context and invite them to the survey.  Remember, they’re doing us a favor.  At the end of the survey, there should be a clear and distinct thank you message, especially if this is a survey going to your own customers.  I’m stunned at how abrupt many surveys end.  If I’m a customer and I’ve just given you 5, 10, or maybe even 15 minutes of my time to answer your survey, and it simply ends at the last screen, that’s not really very nice. 

Here’s some possible text:  “Thank you. Your opinions are very important and will help us to improve our products and services.”  Or, “Thank you. Your input has been extremely valuable.  Stay tuned to our company newsletter to hear how we’ll be applying these important research results.”  Let them know that it wasn’t just an academic exercise; that you plan to actually use the research. 

While it is great to see that there are so many free and low‑cost survey tools available today, such as Ask Your Target Market, SurveyGizmo, QuestionPro, SurveyMonkey, and Zoomerang, there are lots of mistakes that people can easily make when writing surveys.  Finding a great tool may be easy these days, but writing a great survey is not. 

Planning to write a questionnaire? Let Research Rockstar show you how to manage the process and avoid common mistakes. Click here for details.

If you are tracking customer satisfaction at regular intervals, say quarterly or monthly, you may have found that your colleagues want explanations for every increase or decrease in scores—even minor ones. Do the latest results show slight customer satisfaction improvement? If so, they want to know why. If the latest results show a down trend, they want to know why.

In some organizations, I find that people are quick to congratulate themselves on improvements, but willing to dismiss declines as possible “blips.” In other organizations, the culture seems to predisposition people to just the opposite: caution regarding positive news, and anguish to bad.

If you are new to managing such projects, here are some ways to handle those prone to such extremes:

• Remind them that you are tracking a trend. Especially during the first few measurement periods, we have to be cautious about drawing any hard conclusions. It may take a few measurements before you know what kind of “blip” is noise, versus a true increase or decrease.
• Be sure you are aware of contextual phenomenon. Minor fluctuations are often found to be due to things such as awareness of recent stock price performance changes, temporary events (recent marketing campaign halo effect), competitor news, and organizational changes. In some organizations, satisfaction scores can even be seasonal!
• Offer follow-up interviews. In-depth interviews (IDIs) with a subset of survey participants can be a great way to explore hypotheses you and your colleagues may have about certain results.

Customer Satisfaction Surveys That Don’t Satisfy

Ultimately, if you find it hard to manage how colleagues interpret customer satisfaction research results, it may be a clue that the survey design needs improvement. Does it include one or two open-ended questions to capture unscripted customer feedback? Does it capture specific types of customer experiences so you can see how they predict satisfaction levels? Are you capturing both satisfaction attitudes and loyalty behaviors?

We know that customer satisfaction is important, but we also know there is no one-size-fits-all approach.   Different researchers approach it differently which is appropriate—companies in different industries, with different types of client bases, do need different approaches.

Is That a Blip In Our Data, Or Are They Really Happy to See Us?

If you are new to measuring customer satisfaction, it is important to design the survey with an eye towards what types of data your internal audience will find most useful—and to be prepared to address the inevitable questions about upward or downward shifts. Whenever possible, do work with a market research professional experienced in measuring satisfaction in different industries—they will be able to advise you on how to design the survey and interpret the results.

Planning to hire a market research agency? Check out our online class on how to do it.

1. We all have non-researcher friends, colleagues and acquaintances who are writing their own online surveys using one of the many free or low-cost tools.
2. Despite all of the well-publicized voices shouting, “The survey is dead!”, “DIY” survey volume continues to surge.
3. We may tire of hearing, “The survey is dead!”, but we all know that the role of the survey isn’t what it used to be, and we really aren’t that sad about it. Indeed, many of us now spend more time on non-survey methods, either to develop best practices or for “live” projects.
4. Short surveys are better surveys.
5. New tools and technologies create possibilities for survey design and data analysis that never existed before.

So, what does this hodgepodge of facts mean?

The role of the survey is changing. Who creates them, how they are designed, when they are used, and their role relative to other methods is all shifting, fast. To remain effective in this new reality, survey research best practices must change, and the time is now.

New Rules of Survey Design

Some of the New Rules are driven by length, others by what is feasible given recent advances.

Then: Three to five screening questions were acceptable.

Now : One or two. Better yet, none—just collect the profiling data from your MROC database or panel provider.

Then: 15 to 20 minutes average duration was the old “short” compared to the 40 to 60 minute monsters of the old days.

Now: Ten minutes or less.

Then: Multiple questions per screen to minimize number of total screens.

Now: Shorter surveys make it worthwhile to exploit the visual appeal of one question per screen.

Then: Two to three open-ended questions at most, due to both respondent compliance and analysis headaches.

Now: Wildcard. We still want to avoid demanding too much “work” from our respondents, but these questions have become much easier to analyze. Whenever an open-ended question could replace a question with a long list of answer options, it’s worth a look.

Then: Survey design and programming were done by a professional researcher, going through multiple iterations, often taking days if not weeks.

Now: Basic survey design can be easily done by any motivated professional with the help of training, survey templates, or other support.

Then: Surveys designed with the assumption that they are the primary data source for an overall project.

Now: Surveys designed with the context that the survey is only one source of data, and is likely to be used in combination with multiple data sources.

Then: Surveys designed with the assumption that there would be one mode of data collection (typically phone, online or paper).

Now: Surveys designed for multiple modes of data collection, possibly including mobile.

I am generalizing here, and there are plenty of “buts” and “exceptions.” The reality, however, is that the landscape of survey design has changed. We can’t let yesterday’s best practices hold us back today.

Bottom Line

Those of us who remember spending hours (even days) to craft what can only politely be described as visually bland surveys can be forgiven a secret twinge of envy when we see DIYers creating visually stunning surveys in under an hour. But we have to get over ourselves; new tools are in, and the old rules are out.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

1. Why should I take the time to complete a survey? You should tell them, and be earnest. Let your participants know how much you value their input. Let them know how the data will be used. Key phrases include:

• “Your input will help us to improve our services.”
• “Your feedback will help us develop new products for clients like you.”
• “At company X, we are always seeking new ways to improve customer service. And the best way to do that is to ask the experts—our customers.”

2. What’s in it for me? Offer an incentive. Yes, it helps. The promise of future products that are aligned with their needs is nice, but a bit intangible. If this is a survey of your own customers, offer them a coupon or discount code. Other popular items include Amazon gift cards and drawings for tablet computers. Note that different states and countries have different laws about contests and drawings; be sure to get appropriate legal advice.

3. Is it going to be annoying? Let them know up front that you have a nice, short, respectful survey (and be sure that your survey is just that).

Of course, the invitation text is only part of the puzzle. For more tips on maximizing response rates, see Jeffrey Henning’s excellent article from last March, which cites the work of Weimiao Fan and Zheng Yan, and another one from May citing the work of Joel David Bloom, Ph.D., from the University at Albany .

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Then please join me for a practical, fast class on June 9th, in Waltham MA. In just under three hours, attendees will learn the basics of planning, designing and writing great online surveys.

This is a small class format, so everyone will be able to ask questions and get answers. All attendees will receive a class workbook and sample survey designs.

Click below for details!

Research Rockstar Survey Design Class June 9