Cathy’s point, “Customer satisfaction surveys are for measuring, not intervening.”

And Jeffrey’s, “Follow up with every dissatisfied customer who takes a survey.”

To be frank, my opinion on this topic has changed in just the past year or two. Before then, I was an ardent believer that all research must by anonymous—no matter what. I felt that any direct follow-up would show research participants that their survey responses could result in unexpected communications—and even if “helpful”, this experience could still impact future willingness to participate in research.

But in the past couple of years, two things have happened:

• First, I have been working with many clients who need to show that market research is not an academic exercise. Who need to demonstrate that research can directly, immediately, have positive outcomes. Many client-side market researchers have to negotiate for budget with non-researchers, who often view such studies as nice, but not necessarily actionable. Imposing anonymity on customer feedback reduces the research’s potential for clear, measurable usefulness.
• Second, I have seen raw data from several studies where it was obvious that participants expected follow-up. Indeed, anyone who has done a customer satisfaction survey knows that open-ended questions will often return entries such as, “The last software upgrade didn’t work—can you please fix it?” or “I have called your customer service number twice and can’t reach a live human being!”  You can bet that if they take the time to type that into a survey and you don’t follow-up, the damage will be irreparable.

Anonymity in Market Research

Yes, most surveys should be anonymous. But customer satisfaction surveys are an exception. Make it clear at the beginning or end of the survey that respondents can opt out (or opt in, if you prefer) of follow-up. Provide a phone number, web site or email address that can be used for any questions about how responses will be used.  The reality is that most customers expect follow-up.

What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.

Want to learn more about customer satisfaction research? Check out the Research Rockstar class here: ClassList.

After all, you may very well have a list of customers that would be convenient for you to use for your survey project. But is that a wise choice? Maybe, maybe not.

In the quest for objectivity, we often do research with a mix of customer and non-customers. At the end of the project, we compare their responses to answer questions like:

• Do our current customers perceive our brand differently than do non-customers?
• Do our current customers value our product category differently than non-customers?
• Do the two groups differ in terms of feature requirements? Price sensitivity? Other purchase behavior?

Clearly, there is a lot of value in surveying both current customers and “potential” customers.  But realistically, it can be a lot easier and faster to survey current customers. After all, you may have their contact information so emailing them a survey invitation is a snap. They know who you are, and may even like you enough to want to help out by sharing their opinions—so response rates are good. And you know they are real—no worries about the veracity of responses.

In contrast, getting access to non-customers can be expensive, time-consuming, and risky. Sure you can buy sample from a panel provider, or seek other options (partnerships with professional associations, user groups, etc that may meet your needs). But how much will it cost? How long to arrange it? What kind of response rate will you get? How are those respondents authenticated?

So, what’s a time-constrained, budget-sensitive researcher to do?

In an ideal worked, you would examine your research objective, and then decide if it is best met by surveying current customers only, or a mix. For example, if your objective is to choose which features need to be in the next release of a software product, you may very well want both groups. Current customers, to keep them happy and maximize their likelihood to upgrade to the new release when it is available. New customers, in hopes of attracting them to the product for the first time.

But if time or budget doesn’t allow reaching non-customers?

Then you need to ask yourself two questions:

1. Are my customers an acceptable proxy for non-customers? Do I know enough about both groups to understand how different they are—to have an informed opinion of the likelihood that they have similar attitudes and behaviors relevant to the study I am planning? The answers will tell you how much of a risk you are taking, so you can deliver the research with appropriate caveats for anyone who will use it.
2. Is there an acceptable fallback? Even 10 interviews with non-customers is better than nothing. Or perhaps an outside expert with relevant experience is available (try Zintro or GLG for quick access to expert networks) for an objective sanity check.

Customer insights gathered through research deliver great value. But we always need to think about what we really mean by “customer”—current customers, or potential ones? Even if time or financial constraints prohibit you from conducting research with the ideal mix, take the time to understand the risks or consider fallback options.

[Subscribe by email to get the latest blogs as they are published. Click here:  Yes, Send me Blog posts by email!]

The following topics are being scheduled:

• Introduction to Market Research Project Types & Methods (2 hours)
• Introduction to Market Research Project Management (2 hours)
• Product Concept Testing (1.5 hours)
• Designing Online Questionnaires (2 hours)
• Divide & Conquer: Introduction to Market Segmentation (1.5 hours)
• Creating Your Market Research Strategic Plan (1.5 hours)

Interested? Fill out this short form and you’ll get date and location information:  Seminars.  As soon as details are confirmed, you’ll get an invitation.

Thanks!

• Quality sample.  Having quality sample is critical. Avoid sources that can’t talk credibly about their recruitment tactics or list sources. You don’t want to bake a cake with spoiled eggs.
• Sufficient sample quantity. If you are purchasing outside sample, be sure you have access to enough—or at least have a fallback strategy in place in case your incidence rates are tragically low.  I have seen many otherwise brilliantly managed projects experience dramatic schedule slips because of poor sample quantity planning.
• Expert research design. Don’t sabotage a project by fielding a poorly constructed questionnaire or applying an inappropriate methodology for the given objective.  Even if you intend to do a project in-house, getting help in this area will pay for itself by avoiding wasted time and money.
• Competent data analysis. If you spend the time and money to collect beautiful data, don’t skimp on the analysis. This can be harder than it sounds, especially for projects that have had data collection problems. In these cases, you can become so exhausted with the project that by the time data analysis is due, you are just ready to be done with it. Resist the temptation or get help.  Skipping on data analysis can backfire in 2 ways. First, someone else will look at the data and find an important result that you missed. Ouch. Second, you won’t get a good return on the research investment—leaving a bad impression that may derail you the next time you try to get budget for a market research study.
• Decent reporting.  Rushing to get the research delivered is understandable—but unfortunate typos, mislabeled charts or poor writing can turn off the audience and damage the research’s credibility.  Yes, really.

If you have people with these skills in-house—great, do it in-house. If you want to augment with outside expertise for discrete tasks, there are many boutique agencies and market research freelancers available for such work. And if you want an experienced expert to run the project for you, by all means hire a market research agency. But don’t skimp—it always backfires.

[What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.]

Unintentional misuse of market research is best avoided through effective quality assurance processes and well-documented (and communicated) methodologies.  Making sure report readers understand what types of conclusions are appropriate to draw from a research effort, and making sure results are reported in a precise fashion, are both key parts of managing any market research project.

The harder challenge is preventing intentional misuse.

This is truly one of the biggest challenges for market research managers—preventing clients (internal or external ones) from purposefully manipulating results for self-serving purposes. But wait, is it the market research manager’s job to keep clients honest? Yes, at minimum, to point out inappropriate use.

What’s a market research project manager to do? Here are 2 tips to prevent market research abuse:

1. Find an executive advocate for the research. Having someone from outside the research function endorse the research does two powerful things. First, it sends a message that the research has value. After all, if an executive is taking the time to digest the results and be vocal about how they plan to use it, it must be important. Second, if the executive is associated with key conclusions from the research, it becomes harder for others to manipulate the results. It doesn’t prohibit the possibility of a constructive debate about how to interpret key findings—which is always to be encouraged. It just minimizes the risk that someone will hijack the project and make inappropriately self-serving conclusions from the data.
2. Include others in the analysis process. If you know the research is at risk of being abused, make a preemptive move by inviting a team of people to participate in the analysis process. If it’s a quantitative research project, host a meeting where you share a selection of key charts and facilitate an open discussion about their implications. If it’s a qualitative study, you can invite a group to view some video highlights or to receive a preview of selected verbatim quotes; again, to facilitate a group discussion. Including 4 to 8 people in an open discussion about research results and what they mean for the organization is a great way to keep everyone honest—and has the added benefit of promoting the research.

What do you think? Do you agree? Have a different perspective? Please add your comment here, or call the blog comments line at 508.691.6004 ext 703. Thanks!

But when comments I share that are “pending moderator review” never appear, it really annoys me.

About 10 days ago, I read an interesting article on TMCnet—a site that I like for technology-related topics. But it just so happened that this article had some important omissions. So I posted a thoughtful reply. Nothing incendiary. Nothing rude. Just a friendly sharing of information with the author and fellow readers.

It never appeared.

After a week, I emailed the editor. Still nothing.

The original article recommends NPS (Net Promoter Score) as the optimal standard for customer satisfaction with telecommunications providers. Ummm, no. So since I didn’t get to share on the TMCnet site, let me share some information here for those of you interested in measuring customer satisfaction in the telecommunications space.

• “There are many scenarios in which customers may be satisfied with certain service levels or offerings yet refrain from recommending or referring the larger offering to their friends.” Yes, this is very true.
• “…customer referrals – should be the ultimate measure of customer satisfaction and should be cultivated to the greatest extent possible.” Not necessarily.

In telecommunications, willingness to refer is not always the best metric. Having done over a hundred research studies on telecomm topics over the past 20+ years, I know that other items can be more relevant. For example, two items that are very important in the telecomm space:

1. Willingness to renew (vs. propensity to brand switch). For some service providers, lack of brand loyalty is a huge challenge. And cost of customer acquisition can be quite high. So for them, the most useful metric can be renewal intent.
2. Interest in “add-ons” (incremental features/services that would increase $/customer). Again, because the cost of customer acquisition can be high in telecomm, some service providers focus not only on retention but on extensions; how can we sell more to the existing customer base? That’s why in telecomm you often hear people talk about raising ARPU (average revenue per user). And customers’ willingness to buy more says a lot (like how well the proposed add-ons align with their interests, and how far the brand has permission to extend).

Yes, NPS is a wonderfully efficient approach to measuring customer loyalty. But it isn’t the only one. Customer satisfaction and loyalty research is not a one-size-fits all proposition. Telecomm providers need to take the time to identify the best metrics for their research to be truly useful.

[As always, please add a comment or question here, or call the Blog Requests line (508.691.6004). Thanks!]

That’s right. Two.

There’s the stated research objective. Perhaps something likes, “Determine current levels of brand awareness in 5 key market areas,” or “Test 6 potential new marketing messages for alignment with emerging customer needs.”

Then there’s the other, assumed objective: getting engaged respondent participation. This is an implicit objective that too often gets minimized. Yes, we all know we have to do everything we can to maximize response rates, but the choice of methodology is too often driven by the research objective—not the respondents’ needs.

As researchers, we talk a lot about matching the methodology to the first objective. But given low response rates and the preciousness of qualified respondents, we need to focus a lot more on matching the methodology to the audience.

An Example

A researcher I know from a software company was upset after working with a market research agency on a huge study of IT executives. They collected over a thousand responses to an online survey, but data collection was brutally slow due to low response rates. When she finally got the data, she had a lot of important items to which there were a surprising percent of neutral or even “don’t know” responses. Putting aside that this issue should have been caught during the pre-test phase of the project, this was hugely disappointing.

I looked over the screening criteria myself, just to see what the scoop was, and it was obvious that the audience they were targeting was too senior for the 25-minute, very technical, online survey. The topic was about a fairly new technology, so chances are they were interested in the topic—but the methodology choice and level of detail was wrong.

The Shadow Objective

It’s always there. The need to match the project’s methodology with the target respondents’ preferences and behaviors. Maybe you want quantitative data, but the target group gets too many similar requests as is. Maybe you want to do focus groups, but your target population works in a field where scheduling is too uncertain for them to commit to 2 hours of time. Maybe you want to do a phone survey, but your audience has a low penetration of landlines.

Bottom line

Choosing the best methodology for any research study requires considering the project’s objective and the shadow objective. The good news? These days there are so many methods and tools that can make the research experience engaging, there is no need to be constricted by the choice of survey versus focus group.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

If you allow employees to invite customers to take customer satisfaction feedback surveys, make sure they aren’t saying things like, “The highest score is a 10. I hope I earned a 10!” You have probably experienced this yourself, perhaps at an auto dealership or retail chain. Upon completion of your transaction, the clerk give you the feedback survey instructions (perhaps advising of a phone call you will receive, or giving you a URL to use), and then says something like, “I’m hoping you rate me as “extremely helpful.”

Obviously, such behavior not only taints the data, it can also make customers uncomfortable.

So how can you determine if your customer feedback system is being abused? Here are three easy options:

• Simple analysis. For example, if you are in retail you may be able to run the statistics on individual sales people or customer service reps to see if their scores appear artificially high or simply too consistent.
• Get outside help.  Deploy some mystery shoppers on a discovery mission. What do they experience?
• Ask. In the feedback survey itself, ask respondents if anyone told them how to respond or suggested a desirable feedback score.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

If you do an annual employee survey to gather current perceptions and attitudes—great! This is an area of research that can uncover real opportunities to boost employee satisfaction and productivity, and even identify ways to save money. But do be sure to include technology-related content—especially if your employees use computers, communications and related technologies as part of their jobs.

Not sure where to start? Take an hour to brainstorm some tech-related hypotheses. Before you know it, you will have a good set of items worthy of investigation. Here are some examples that are relevant to many organizations today:

• Our employees dislike using their desk phones; they prefer to forward their calls to their mobile phones
• Our employees are not following back-up protocols consistently
• Employees need more training on new applications
• Employees want to use (insert application or web site) but the company currently blocks it
• Employees want more leniency about what they can choose to install on company-provided computers
• Many people find the current IT help desk processes confusing, so they tend to avoid it until a problem becomes critical
• Employees are running blogs that sometimes touch on business topics, but are unaware of what disclaimers and protocols they should follow

Of course, the point is not to find out what they want and instantly provide it. There are obvious reasons why, for example, certain websites are blocked. Still, keeping an open mind and hearing this feedback may suggest a need to better explain why the policy exists or perhaps even identify situations where exceptions should be made.

Just Because You Build It, Doesn’t Mean They Will Come

Companies spend a lot of money on technology in an attempt to boost employee productivity—but how many go back to make sure it has happened? I have seen cases where employee research has uncovered surprising results—like that employees aren’t even using a telephony feature assumed to be critical, or that they avoid a specific business process because the related application is too cumbersome.

Does adding technology-related content add too much?

If the new content would make your annual employee survey too lengthy or onerous, then consider your options. Perhaps a smaller percentage of employees can be asked to complete the additional questions. Or maybe the employee base can be divided and asked to take 2 different questionnaires. Perhaps the questionnaires can be separated and done 6 months apart—so that employees don’t feel bombarded. Or maybe the tech questions can be tackled through in-house focus groups, for a more qualitative and in-depth discussion.

In any case, as with all employee research, it’s critical to make sure that upon completion, employees are thanked and next steps identified. Communicating how the research is driving specific, concrete actions that will improve the workplace will go a long way to boosting employee satisfaction. In contrast, employees who share thoughtful feedback and then see no action will be less willing the next time you ask.

[Is this a topic of interest? Interested in some help adding IT-related content to your employee research program? Contact me at kkorostoff@ResearchRockstar.com]

It’s great to hear a social media authority praise—even promote—the benefit of asking customers’ about their views. In this case, Peter was making the excellent point that companies need to ask customers how they want their information delivered. At today’s presentation in downtown Boston, Peter specifically advised the business folks in attendance to take the time to ask their audiences (clients, prospects, donors, whatever) how they want to receive information. He points to the fact that the way information is delivered these days is extremely fragmented. Nobody can afford to simply guess how their audience wants to get information.  He points out that simply taking the time to ask people how they want information delivered can save huge amounts of money.

Now granted, he didn’t go beyond that to advise the best ways…or potential challenges…of asking people how they want to receive information. But as someone who has done actual primary research on this for many business clients over the years, I can tack on two bits of advice.

Self-reporting can be way off, so augment with actual behavioral data. If you simply give people a list of information sources (newspapers, magazines, TV, radio, LinkedIn, YouTube, etc) and ask them which are their preferred ones, you will get a very particular picture. If you ask the same people which ones they have used in the past 48 hours, you will get a very different picture. Why? People don’t always like to admit—either to themselves or others—how few sources they use. Consider my Aunt Sally. If you asked Aunt Sally how she gets news, she would emphatically tell you that she reads The New York Times, The Economist, and watches the evening news. Well, what she really means is that she wishes she did, but in reality, most of her news comes from the radio and her Yahoo home page.

Sure, it’s still good to ask people what they want. But balance that with some actual behavioral data as well. What have you read in the past 24 hours? What websites have you visited today? Nice, concrete questions.

Understand the behaviors associated with different information sources. This can be very enlightening. For example, let’s say IT buyer Jack works for a large retail chain. He gets information about IT companies from trade magazines, business magazines, and CNET. But what does he do with the information?  Maybe with the trade magazines he tends to rip out anything of interest and throw the rest into recycling. Business magazines? Perhaps he occasionally forwards an issue to a colleague, but less than once out of every 4 issues. CNET? Turns out he loves the email feature (email a story to a friend), and forwards at least 3 CNET articles a week to his peers—internal and external. So if I wanted to get on Jack’s radar screen and optimize the chance of him sharing info, I now know what to do.

How about you?

So do you know what information sources your target clients actually use? Trust? Share? The topic of information sources is actually pretty complex, and there are lots of other cool research tricks that can be used to understand behaviors. But as Peter Shankman said today, simply finding out what sources people use is a huge step towards optimizing a communications strategy.

[Do you get Research Rockstar blog posts via RSS? If not, here's the link: http://feeds.feedburner.com/ResearchRockstar]