Market Research Training from Research Rockstar

With the summer season upon us, along comes a most welcomed relief for many market research project managers. The workload slows and creates a great opportunity to take care of some market research housekeeping. For survey writers, one of the best uses of slow time is to spend it creating (or updating) standard survey templates for use year-round.

Without the normal crush of deadlines, market researchers can create templates with these goals in mind:

1. Set standards.  Use this time to think carefully about how you want to standardize specific question types, along with formatting and scaling options.
2. Get approvals. Get approval from those colleagues or managers who will be involved in year-round research efforts. Explain you are constructing standard templates and want their input during the slow season—this will help everyone come crunch time.

For those newer to research, you will find that there are three common survey templates that come in handy.  A few examples of question types are included below, but there are many options—so be sure to look at several examples before you craft your templates.

Survey Template #1: Customer Satisfaction Research

A simple satisfaction survey would consist of four or five questions to gauge satisfaction and loyalty. Of course, the type of customer satisfaction and loyalty questions depends on whether you’re selling business-to-business or business-to-consumer, whether you’re selling services or products and so forth. In general, you’re going to want an overall gauge of satisfaction, which commonly uses a five-or seven-point Likert scale question. Add a few follow-up questions about satisfaction relating to the specific aspects of your products or services, as relevant, such as customer service, product’s ease of use, and possibly aesthetics.

As an example of how this might need to be modified for different product categories, let’s consider a snack product company. For this case, satisfaction measures might focus on the variety of flavors offered, response to specific flavors, and package size.

Survey Template #2: Website Feedback

If your organization interacts with customers on its website, it’s good to have a standard template for collecting website feedback. This could be used on either a transaction or a rotation basis (so that customers see it on every 10 or 20 visits), or maybe it’s something you will use once a quarter.

Common questions collect feedback on overall attractiveness, distinctiveness, and ease of use. So answer options might use a scale of “very mundane” to “very exciting”; or maybe a range from “amateurish” to “very professional”; or perhaps, “not at all easy to use” to “very easy to use”.

Other key questions may include:

• “Were you able to find the information you were looking for on our website?”
• “How likely are you to visit this web site again in the next 30 days?” (or whatever timeframe would make sense for your particular category).

Survey Template #3: Customer Service Transaction

You may want to have a survey that’s triggered every time someone completes a support call or other type of customer service transaction with your organization. This could be done through a call center, email, or even through a social media interaction such as on a Facebook fan page or via Twitter.

Typical questions ask about:

• How quickly they got a response, which gives you an objective assessment of whether or not it was timely
• Their satisfaction with the timeliness of the response
• Their satisfaction with the quality of the response

Moreover, also use it as an opportunity to make sure that the matter was completed successfully so that you can create a red flag if necessary

Market Research Planning

In my town, the Department of Public Works knows that we’re going to get a lot of snow each winter, and you can be sure that for the couple of months before winter comes, they’re stockpiling salt and sand for the roads. They’re using that relatively slow period before crunch time hits to make sure they’re prepared.

Well, it’s the same thing for survey writing. Let’s take advantage of this slow time to make sure that we have everything we’re going to need before the next crunch time hits — as it inevitably does.

Click here to check out Research Rockstar’s full line of Online Market Research Training Classes.

If you are tracking customer satisfaction at regular intervals, say quarterly or monthly, you may have found that your colleagues want explanations for every increase or decrease in scores—even minor ones. Do the latest results show slight customer satisfaction improvement? If so, they want to know why. If the latest results show a down trend, they want to know why.

In some organizations, I find that people are quick to congratulate themselves on improvements, but willing to dismiss declines as possible “blips.” In other organizations, the culture seems to predisposition people to just the opposite: caution regarding positive news, and anguish to bad.

If you are new to managing such projects, here are some ways to handle those prone to such extremes:

• Remind them that you are tracking a trend. Especially during the first few measurement periods, we have to be cautious about drawing any hard conclusions. It may take a few measurements before you know what kind of “blip” is noise, versus a true increase or decrease.
• Be sure you are aware of contextual phenomenon. Minor fluctuations are often found to be due to things such as awareness of recent stock price performance changes, temporary events (recent marketing campaign halo effect), competitor news, and organizational changes. In some organizations, satisfaction scores can even be seasonal!
• Offer follow-up interviews. In-depth interviews (IDIs) with a subset of survey participants can be a great way to explore hypotheses you and your colleagues may have about certain results.

Customer Satisfaction Surveys That Don’t Satisfy

Ultimately, if you find it hard to manage how colleagues interpret customer satisfaction research results, it may be a clue that the survey design needs improvement. Does it include one or two open-ended questions to capture unscripted customer feedback? Does it capture specific types of customer experiences so you can see how they predict satisfaction levels? Are you capturing both satisfaction attitudes and loyalty behaviors?

We know that customer satisfaction is important, but we also know there is no one-size-fits-all approach.   Different researchers approach it differently which is appropriate—companies in different industries, with different types of client bases, do need different approaches.

Is That a Blip In Our Data, Or Are They Really Happy to See Us?

If you are new to measuring customer satisfaction, it is important to design the survey with an eye towards what types of data your internal audience will find most useful—and to be prepared to address the inevitable questions about upward or downward shifts. Whenever possible, do work with a market research professional experienced in measuring satisfaction in different industries—they will be able to advise you on how to design the survey and interpret the results.

Planning to hire a market research agency? Check out our online class on how to do it.

Imagine it’s Friday evening. You’ve been scrambling all week, and you’ve decided to unwind with a nice dinner out. The fellow at the desk next to yours has been raving about this new bistro in town—the best steak, perfect wine, and dessert to die for. So you decide to treat yourself, and…

The hostess is rude, the salad limp, the wine warm and the steak unrecognizable. What are the chances you’ll be going back there again? How likely are you to take future restaurant tips from the guy who set you up? Might you even tell a few friends about your horrid experience?

Yes, this has something to do with market research. Or, to be precise, surveys. When bad surveys are circulated, the company that sent them out becomes less trusted. The “consumer” becomes an unhappy customer, and may even tell others about their bad experience—with surveys in general or with the specific company.

WRITING GREAT SURVEYS IS IMPORTANT TO EVERYONE

Anybody who’s had a bad survey experience is likely to have a tainted perception of the process, and that can come out in a number of ways:

1. Participation: They may be less likely to take surveys in the future.
2. Attitude: They might spread negative word of mouth about the company that sent the survey or about market research as a whole.
3. Skepticism: They may be skeptical the next time they see market research results.
4. Investment. If they are business professionals, they may be less supportive of their organization’s investments in market research because they just don’t trust the process.

BEING A GOOD CITIZEN

As good citizens of the market research community, we have an obligation to make sure the questionnaires our organizations are distributing are impeccable. Even those coming from the well-intentioned but usually untrained DIYers. The challenge for many managers is the huge number of colleagues now using low-cost tools for creating surveys. An opportunity, yes. But without proper training and oversight, the chance of creating a bad survey is greater than ever. Here are some low-cost options to help avoid that:

• Examples – Provide a template library of commonly used, approved questions. Demographic questions at minimum, so that your colleagues will be collecting consistent profiling information but avoiding questions that may be too intrusive, or too vague to be useful.
• References – There are some great books out there. The “Handbook of Marketing Scales” by Bearden, Netemeyer and Haws (Sage Publications, 2011) is one favorite. A little technical, but absolutely readable to anyone willing to make an effort.
• Quality Assurance – Appoint one or two people as the Survey Q&A Reviewers, and give them the responsibility (and authority) of sanity-checking any surveys before going live (especially those being sent to your valued customers). Make sure this role is publicized and endorsed by managers.
• Keep it simple – There are a lot of tools out there, and while it might seem counterintuitive, sometimes you’re better off limiting the choices. Instead of many different question types, limit it to a handful (say, multiple and single choice, and Likert scales) to keep the surveys manageable, and therefore less prone to design abuse.
• Training – Basic skills are important. Start with new employee orientation materials and train your workforce. I’m a little biased here since Research Rockstar is an online training company, but there are also others that offer seminars and webinars, including the MRA, Burke Institute and ESOMAR.

SURVEY QUALITY IS CRITICAL

Survey quality is important to those who create them, those who take them and those who depend on their results. It’s in everyone’s best interest to assure that what the public sees reflects the quality and professionalism of the market research industry. As I’ve mentioned in other blogs, having a few good policies is a great place to start. And it might just earn you a nice dinner out come Friday, too.

[Planning to write a questionnaire? Check out a preview of Research Rockstar's questionnaire design process class.]

During the June 23, AMA MRC TweetOff session with myself, Jeffrey Henning (@JHenning), and Cathy Harrison (@VirtualMR), one topic we debated was the role of anonymity in customer satisfaction surveys.

Cathy’s point, “Customer satisfaction surveys are for measuring, not intervening.”

And Jeffrey’s, “Follow up with every dissatisfied customer who takes a survey.”

To be frank, my opinion on this topic has changed in just the past year or two. Before then, I was an ardent believer that all research must by anonymous—no matter what. I felt that any direct follow-up would show research participants that their survey responses could result in unexpected communications—and even if “helpful”, this experience could still impact future willingness to participate in research.

But in the past couple of years, two things have happened:

• First, I have been working with many clients who need to show that market research is not an academic exercise. Who need to demonstrate that research can directly, immediately, have positive outcomes. Many client-side market researchers have to negotiate for budget with non-researchers, who often view such studies as nice, but not necessarily actionable. Imposing anonymity on customer feedback reduces the research’s potential for clear, measurable usefulness.
• Second, I have seen raw data from several studies where it was obvious that participants expected follow-up. Indeed, anyone who has done a customer satisfaction survey knows that open-ended questions will often return entries such as, “The last software upgrade didn’t work—can you please fix it?” or “I have called your customer service number twice and can’t reach a live human being!”  You can bet that if they take the time to type that into a survey and you don’t follow-up, the damage will be irreparable.

Anonymity in Market Research

Yes, most surveys should be anonymous. But customer satisfaction surveys are an exception. Make it clear at the beginning or end of the survey that respondents can opt out (or opt in, if you prefer) of follow-up. Provide a phone number, web site or email address that can be used for any questions about how responses will be used.  The reality is that most customers expect follow-up.

What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.

Want to learn more about customer satisfaction research? Check out the Research Rockstar class here: ClassList.

Perhaps my favorite thing about reading blogs is that I can have a dialog with the author and fellow readers. Friendly debates or spontaneous collaborations are a lot of fun.

But when comments I share that are “pending moderator review” never appear, it really annoys me.

About 10 days ago, I read an interesting article on TMCnet—a site that I like for technology-related topics. But it just so happened that this article had some important omissions. So I posted a thoughtful reply. Nothing incendiary. Nothing rude. Just a friendly sharing of information with the author and fellow readers.

It never appeared.

After a week, I emailed the editor. Still nothing.

The original article recommends NPS (Net Promoter Score) as the optimal standard for customer satisfaction with telecommunications providers. Ummm, no. So since I didn’t get to share on the TMCnet site, let me share some information here for those of you interested in measuring customer satisfaction in the telecommunications space.

• “There are many scenarios in which customers may be satisfied with certain service levels or offerings yet refrain from recommending or referring the larger offering to their friends.” Yes, this is very true.
• “…customer referrals – should be the ultimate measure of customer satisfaction and should be cultivated to the greatest extent possible.” Not necessarily.

In telecommunications, willingness to refer is not always the best metric. Having done over a hundred research studies on telecomm topics over the past 20+ years, I know that other items can be more relevant. For example, two items that are very important in the telecomm space:

1. Willingness to renew (vs. propensity to brand switch). For some service providers, lack of brand loyalty is a huge challenge. And cost of customer acquisition can be quite high. So for them, the most useful metric can be renewal intent.
2. Interest in “add-ons” (incremental features/services that would increase $/customer). Again, because the cost of customer acquisition can be high in telecomm, some service providers focus not only on retention but on extensions; how can we sell more to the existing customer base? That’s why in telecomm you often hear people talk about raising ARPU (average revenue per user). And customers’ willingness to buy more says a lot (like how well the proposed add-ons align with their interests, and how far the brand has permission to extend).

Yes, NPS is a wonderfully efficient approach to measuring customer loyalty. But it isn’t the only one. Customer satisfaction and loyalty research is not a one-size-fits all proposition. Telecomm providers need to take the time to identify the best metrics for their research to be truly useful.

[As always, please add a comment or question here, or call the Blog Requests line (508.691.6004). Thanks!]

Don’t let employees bias customer feedback results. Or worse, don’t let high-score-seeking employees bully customers into giving them inflated scores.

If you allow employees to invite customers to take customer satisfaction feedback surveys, make sure they aren’t saying things like, “The highest score is a 10. I hope I earned a 10!” You have probably experienced this yourself, perhaps at an auto dealership or retail chain. Upon completion of your transaction, the clerk give you the feedback survey instructions (perhaps advising of a phone call you will receive, or giving you a URL to use), and then says something like, “I’m hoping you rate me as “extremely helpful.”

Obviously, such behavior not only taints the data, it can also make customers uncomfortable.

So how can you determine if your customer feedback system is being abused? Here are three easy options:

• Simple analysis. For example, if you are in retail you may be able to run the statistics on individual sales people or customer service reps to see if their scores appear artificially high or simply too consistent.
• Get outside help.  Deploy some mystery shoppers on a discovery mission. What do they experience?
• Ask. In the feedback survey itself, ask respondents if anyone told them how to respond or suggested a desirable feedback score.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

Word clouds are so cool.  The first time I found Wordle, I played with it for an hour.

But to be honest, I never really thought of it as much more than a curiosity…sort of a twist on trivia. What words are most frequent on a favorite blog’s page? Wordle would show it in cool colors and an appealing mix of horizontal and vertical lay-outs.

So when I saw that Tribe Research‘s Tribal ToolKit now includes a word cloud tool, I was eager to take it for a test drive.  How could word clouds really add insight? Be more than just a toy?

CloudMaker is a word cloud tool that can draw input from web pages, Twitter feeds (though, alas, restricted to Twitter’s self-imposed 20 tweet limit), and CSV files.  The CSV files part is the important one—since this is what allows you to import any text you want…like open-ended responses from a survey you were loathe to tally the hard way.

To get started, I set up an account (you need to buy some very reasonably-priced credits). Once I had my account, my first use was to check my blog page.  Was my content well-aligned with my intended message? Well, let’s use CloudMaker to find out. Here is the result (I could have added colors but I chose to keep it simple):

(I am not sure why it is pasting in sort of fuzzy; the original image looks much better on my computer).

The process was easy. With no training, it took me 6 minutes to specify the web page from which to draw the blog entries, and set some parameters. For example, I only wanted words that appeared at least 5 times to be included in the cloud.  I also spent a couple of minutes blocking common words (“you,” “your,” “some,” “very,” etc.).

The real power will be for people with open-ended responses to analyze.  Simply create a text file (easy to export from SPSS), and upload it to CloudMaker. You can even opt to have CloudMaker include the frequency counts. I’d also like to try this for analyzing interview transcripts.

Small is Beautiful: The 1-question Survey

There is a survey option as well. Want to know what 3 words people associate with your product, service, or overall brand?  The survey tool in CloudMaker lets you create a 1 question poll that accepts up to 3 text answers (best for 1 or 2-word answers). You get a URL to forward to survey-takers, and the software has built-in security (like preventing the same computer from answering the survey more than once). One catch: write your question carefully—once you create it and start data collection, you can’t edit it. So if you want to pre-test, plan accordingly (either do the pre-test via an email version of the question, or plan to use your CloudMaker credits for a pre-test).

Overall, Tribe Research has created an easy tool for applying word clouds to real business.

[I welcome any and all comments! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 10/23.  Our last winner was Joy Levin—Congrats Joy!!]

I recently had the opportunity to serve as a guest speaker for a Vovici webinar titled, “It’s Greek to Me: Multilingual Surveys.” It’s a great topic, and one that researchers gloss over at their own peril.

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]

Are you planning your first Customer Satisfaction project? If so, please take a moment and check all that apply (well, mentally anyway):

– I feel confused by all of the different methodology recommendations I am getting
– I feel a bit stunned by some of the proposed budgets
– I am wondering how my organization will actually use the results when the project is done
– I am worried that I don’t know enough to anticipate likely roadblocks

If some or all of these statements apply to you, don’t worry. These are perfectly common concerns, and just show that you are thinking carefully about the realities of conducting a customer satisfaction project.

So to help you get started, I’ll address some of these items briefly.

What methodology will be best?

There is no one-size-fits-all approach. Lots of methods can be used (including various qualitative and quantitative ones).  Your best mix will be selected based on:

• How you plan to use the research results. For example, a design meant to provide inputs to executive bonus calculations will be different than one designed primarily to inform organizational performance goals. In fact, any consultant or sales person who tells you their approach is the only legitimate one is only interested in sales—not insights.
• Your customer base. Its size, profile and geographic distribution all come into play. A customer sat (and yes, that is the vernacular) project designed for a consumer goods manufacturer in a highly competitive market with millions of customers worldwide will not be the same as one for a B2B software company with a primarily US-based client base of 500.
• Your budget. Yes, I said it. Any customer sat project can be designed (or, over-designed) to the tune of hundreds of thousands or even millions of dollars. But maybe your budget is limited. If your budget is $50,000, or even $20,000, options exist. Not all research firms will be interested (some are operationally optimized for larger studies), but plenty will be. Be honest about your budget limits and you will get appropriate proposals.

How will my organization use the final results?

The most common real-world uses of customer sat data are:

• To set and track organizational performance goals. For example, a company may have as a stated goal, “80% of customers report they will buy from us again in the next 6 months.” (Of course, just because 80% say they will, doesn’t mean 80% will in reality. Still, the fact they think they will is important).
• To generate inputs for executive bonus calculations.
• To identify opportunities for innovation (sometimes based on areas of customer dissatisfaction)
• To identify which customer touch points most directly predict loyalty behaviors (such as repeat purchasing and positive word of mouth); this creates a prioritized list of areas for improvement, optimized for the most impact.

What roadblocks will I likely encounter?

Short answer: a bunch. But the single most common issue is sample source. Do you really have a list of customers you can recruit to participate? Do you have other direct access to qualified customers? If you do, great. But a surprising number of companies do not.  I have worked with some really big clients, and some have had pitifully poor customer lists.  And since this is a customer sat project, you do need access to valid, qualified customers.

Here are 2 problems I have seen many times each:

• B2B companies that realize their lists a) are out of date (more than 10% of the names/phone numbers/email addresses are incorrect) b) have bad contact info (the customer sat survey needs to be of people who use the product, but the list is comprised of purchasing agents who buy, but do not use). Result? Significant delays and budget overruns.
• Consumer companies that overestimate the feasibility of using purchased lists to reach their customers. The result? Significant budget overruns.

Want More?

If you’d like a little more of an introduction to planning your first customer sat research project, please check out this 10 minute video on YouTube: Research Rockstar on YouTube.

Or, for a 53-minute, comprehensive introduction, check out this link to the Research Rockstar store: Store Link.

[Any questions or comments? Leave them here or call the blog requests line at 508.691.6004. Thanks!]