Cathy’s point, “Customer satisfaction surveys are for measuring, not intervening.”

And Jeffrey’s, “Follow up with every dissatisfied customer who takes a survey.”

To be frank, my opinion on this topic has changed in just the past year or two. Before then, I was an ardent believer that all research must by anonymous—no matter what. I felt that any direct follow-up would show research participants that their survey responses could result in unexpected communications—and even if “helpful”, this experience could still impact future willingness to participate in research.

But in the past couple of years, two things have happened:

• First, I have been working with many clients who need to show that market research is not an academic exercise. Who need to demonstrate that research can directly, immediately, have positive outcomes. Many client-side market researchers have to negotiate for budget with non-researchers, who often view such studies as nice, but not necessarily actionable. Imposing anonymity on customer feedback reduces the research’s potential for clear, measurable usefulness.
• Second, I have seen raw data from several studies where it was obvious that participants expected follow-up. Indeed, anyone who has done a customer satisfaction survey knows that open-ended questions will often return entries such as, “The last software upgrade didn’t work—can you please fix it?” or “I have called your customer service number twice and can’t reach a live human being!”  You can bet that if they take the time to type that into a survey and you don’t follow-up, the damage will be irreparable.

Anonymity in Market Research

Yes, most surveys should be anonymous. But customer satisfaction surveys are an exception. Make it clear at the beginning or end of the survey that respondents can opt out (or opt in, if you prefer) of follow-up. Provide a phone number, web site or email address that can be used for any questions about how responses will be used.  The reality is that most customers expect follow-up.

What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.

Want to learn more about customer satisfaction research? Check out the Research Rockstar class here: ClassList.

But when comments I share that are “pending moderator review” never appear, it really annoys me.

About 10 days ago, I read an interesting article on TMCnet—a site that I like for technology-related topics. But it just so happened that this article had some important omissions. So I posted a thoughtful reply. Nothing incendiary. Nothing rude. Just a friendly sharing of information with the author and fellow readers.

It never appeared.

After a week, I emailed the editor. Still nothing.

The original article recommends NPS (Net Promoter Score) as the optimal standard for customer satisfaction with telecommunications providers. Ummm, no. So since I didn’t get to share on the TMCnet site, let me share some information here for those of you interested in measuring customer satisfaction in the telecommunications space.

• “There are many scenarios in which customers may be satisfied with certain service levels or offerings yet refrain from recommending or referring the larger offering to their friends.” Yes, this is very true.
• “…customer referrals – should be the ultimate measure of customer satisfaction and should be cultivated to the greatest extent possible.” Not necessarily.

In telecommunications, willingness to refer is not always the best metric. Having done over a hundred research studies on telecomm topics over the past 20+ years, I know that other items can be more relevant. For example, two items that are very important in the telecomm space:

1. Willingness to renew (vs. propensity to brand switch). For some service providers, lack of brand loyalty is a huge challenge. And cost of customer acquisition can be quite high. So for them, the most useful metric can be renewal intent.
2. Interest in “add-ons” (incremental features/services that would increase $/customer). Again, because the cost of customer acquisition can be high in telecomm, some service providers focus not only on retention but on extensions; how can we sell more to the existing customer base? That’s why in telecomm you often hear people talk about raising ARPU (average revenue per user). And customers’ willingness to buy more says a lot (like how well the proposed add-ons align with their interests, and how far the brand has permission to extend).

Yes, NPS is a wonderfully efficient approach to measuring customer loyalty. But it isn’t the only one. Customer satisfaction and loyalty research is not a one-size-fits all proposition. Telecomm providers need to take the time to identify the best metrics for their research to be truly useful.

[As always, please add a comment or question here, or call the Blog Requests line (508.691.6004). Thanks!]

If you allow employees to invite customers to take customer satisfaction feedback surveys, make sure they aren’t saying things like, “The highest score is a 10. I hope I earned a 10!” You have probably experienced this yourself, perhaps at an auto dealership or retail chain. Upon completion of your transaction, the clerk give you the feedback survey instructions (perhaps advising of a phone call you will receive, or giving you a URL to use), and then says something like, “I’m hoping you rate me as “extremely helpful.”

Obviously, such behavior not only taints the data, it can also make customers uncomfortable.

So how can you determine if your customer feedback system is being abused? Here are three easy options:

• Simple analysis. For example, if you are in retail you may be able to run the statistics on individual sales people or customer service reps to see if their scores appear artificially high or simply too consistent.
• Get outside help.  Deploy some mystery shoppers on a discovery mission. What do they experience?
• Ask. In the feedback survey itself, ask respondents if anyone told them how to respond or suggested a desirable feedback score.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

But to be honest, I never really thought of it as much more than a curiosity…sort of a twist on trivia. What words are most frequent on a favorite blog’s page? Wordle would show it in cool colors and an appealing mix of horizontal and vertical lay-outs.

So when I saw that Tribe Research’s Tribal ToolKit now includes a word cloud tool, I was eager to take it for a test drive.  How could word clouds really add insight? Be more than just a toy?

CloudMaker is a word cloud tool that can draw input from web pages, Twitter feeds (though, alas, restricted to Twitter’s self-imposed 20 tweet limit), and CSV files.  The CSV files part is the important one—since this is what allows you to import any text you want…like open-ended responses from a survey you were loathe to tally the hard way.

To get started, I set up an account (you need to buy some very reasonably-priced credits). Once I had my account, my first use was to check my blog page.  Was my content well-aligned with my intended message? Well, let’s use CloudMaker to find out. Here is the result (I could have added colors but I chose to keep it simple):

(I am not sure why it is pasting in sort of fuzzy; the original image looks much better on my computer).

The process was easy. With no training, it took me 6 minutes to specify the web page from which to draw the blog entries, and set some parameters. For example, I only wanted words that appeared at least 5 times to be included in the cloud.  I also spent a couple of minutes blocking common words (“you,” “your,” “some,” “very,” etc.).

The real power will be for people with open-ended responses to analyze.  Simply create a text file (easy to export from SPSS), and upload it to CloudMaker. You can even opt to have CloudMaker include the frequency counts. I’d also like to try this for analyzing interview transcripts.

Small is Beautiful: The 1-question Survey

There is a survey option as well. Want to know what 3 words people associate with your product, service, or overall brand?  The survey tool in CloudMaker lets you create a 1 question poll that accepts up to 3 text answers (best for 1 or 2-word answers). You get a URL to forward to survey-takers, and the software has built-in security (like preventing the same computer from answering the survey more than once). One catch: write your question carefully—once you create it and start data collection, you can’t edit it. So if you want to pre-test, plan accordingly (either do the pre-test via an email version of the question, or plan to use your CloudMaker credits for a pre-test).

Overall, Tribe Research has created an easy tool for applying word clouds to real business.

[I welcome any and all comments! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 10/23.  Our last winner was Joy Levin—Congrats Joy!!]

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]

– I feel confused by all of the different methodology recommendations I am getting
– I feel a bit stunned by some of the proposed budgets
– I am wondering how my organization will actually use the results when the project is done
– I am worried that I don’t know enough to anticipate likely roadblocks

If some or all of these statements apply to you, don’t worry. These are perfectly common concerns, and just show that you are thinking carefully about the realities of conducting a customer satisfaction project.

So to help you get started, I’ll address some of these items briefly.

What methodology will be best?

There is no one-size-fits-all approach. Lots of methods can be used (including various qualitative and quantitative ones).  Your best mix will be selected based on:

• How you plan to use the research results. For example, a design meant to provide inputs to executive bonus calculations will be different than one designed primarily to inform organizational performance goals. In fact, any consultant or sales person who tells you their approach is the only legitimate one is only interested in sales—not insights.
• Your customer base. Its size, profile and geographic distribution all come into play. A customer sat (and yes, that is the vernacular) project designed for a consumer goods manufacturer in a highly competitive market with millions of customers worldwide will not be the same as one for a B2B software company with a primarily US-based client base of 500.
• Your budget. Yes, I said it. Any customer sat project can be designed (or, over-designed) to the tune of hundreds of thousands or even millions of dollars. But maybe your budget is limited. If your budget is $50,000, or even $20,000, options exist. Not all research firms will be interested (some are operationally optimized for larger studies), but plenty will be. Be honest about your budget limits and you will get appropriate proposals.

How will my organization use the final results?

The most common real-world uses of customer sat data are:

• To set and track organizational performance goals. For example, a company may have as a stated goal, “80% of customers report they will buy from us again in the next 6 months.” (Of course, just because 80% say they will, doesn’t mean 80% will in reality. Still, the fact they think they will is important).
• To generate inputs for executive bonus calculations.
• To identify opportunities for innovation (sometimes based on areas of customer dissatisfaction)
• To identify which customer touch points most directly predict loyalty behaviors (such as repeat purchasing and positive word of mouth); this creates a prioritized list of areas for improvement, optimized for the most impact.

What roadblocks will I likely encounter?

Short answer: a bunch. But the single most common issue is sample source. Do you really have a list of customers you can recruit to participate? Do you have other direct access to qualified customers? If you do, great. But a surprising number of companies do not.  I have worked with some really big clients, and some have had pitifully poor customer lists.  And since this is a customer sat project, you do need access to valid, qualified customers.

Here are 2 problems I have seen many times each:

• B2B companies that realize their lists a) are out of date (more than 10% of the names/phone numbers/email addresses are incorrect) b) have bad contact info (the customer sat survey needs to be of people who use the product, but the list is comprised of purchasing agents who buy, but do not use). Result? Significant delays and budget overruns.
• Consumer companies that overestimate the feasibility of using purchased lists to reach their customers. The result? Significant budget overruns.

Want More?

If you’d like a little more of an introduction to planning your first customer sat research project, please check out this 10 minute video on YouTube: Research Rockstar on YouTube.

Or, for a 53-minute, comprehensive introduction, check out this link to the Research Rockstar store: Store Link.

[Any questions or comments? Leave them here or call the blog requests line at 508.691.6004. Thanks!]