Have you ever suspected that you, or your colleagues, over-rely on the ubiquitous 5‑point scales when designing questionnaires?

There are a lot of cases where what you are trying to capture, whether it is customer attitudes or behaviors, would be best served using a different type of scale or answer option.  For anyone who designs online surveys and is interested in making sure that they are collecting data that is both objective and complete, a good hard look at how you choose your answer options and scales is critical.

In our new class, “Ask It Right: Choosing Answer Types & Scales”, Research Rockstar covers Likert Scales, Semantic Differential Scales, Constant Sum, Ranking Questions, Rating Questions and more.  We specifically highlight examples of how to structure answer options for common research market needs, including customer satisfaction, product concept testing and more.

The class also includes several examples of “bad” questions, and how they can be rewritten to be better.

In market research, we all know we need to make sure that we are collecting data that is going to give us a complete, objective view of a target population’s, attitudes and behaviors. As important as it is to word the questions carefully, it’s equally important to make sure answer options are going to make sense.

Research Rockstar VIP, Backstage and Tour Bus members all have access to the class now; please just visit your member’s pages. Not currently a VIP, Backstage Pass or Tour Bus member?  The class is available for individual purchase here.

We particularly recommend this class to anybody who has colleagues who are new to questionnaire design. This is a very practical class that will help them avoid the common mistakes made by novice market researchers.

Part of being a good market researcher is the ability to determine when the research conclusions we start to draw reflect reality, and when they’re only coincidences. A few shiny data points can easily tempt us to jump to conclusions too quickly—before we dig deeper.

I’m reminded of raising a teenager, who reported that she “just had to have” a certain pair of shoes. They were stylish. They were unbelievably comfortable.  And all her friends were getting them. In good parental fashion, I struck a deal. We’d wait a month until report cards, and if her grades met expectations, we’d consider the purchase of $100 shoes.

A month later the grades were fine, but the shoes were no longer desired. Turns out nobody was wearing them anymore because they were too tight, fell apart and looked funny. Yet had we gone with her initial “analysis”, we’d likely have concluded that this was a great product — and we’d have been wrong. So how do we avoid the trap?

Market Research & Insight

As researchers we have to keep ourselves honest.  A few coincidences do not a key finding make, and I’ve seen some frightening cases of “conclusions” based on weak data analysis and flimsy proof points. This isn’t value—it’s hyperbole.

During analysis we can’t assume that “interesting patterns” reflect a broader reality, nor force a conclusion where none is justified.

Here are two best practices to help practice the discipline of meaningful insight creation:

Challenge your own conclusions

If you’re doing a research study and the data seems to perfectly support all your hypotheses, you need to ask yourself whether your research or the process was biased.  Perhaps your research was intended to confirm things your organization already knew. But if you’re hoping for some real “aha” points, this may not be what you want.  If your research results wrap up too neatly, look closer.  Is there an opportunity to challenge any of these results?  I’ve done several customer loyalty studies that at face value gave very positive results—customers were broadly satisfied and loyal. Digging deeper, though, we learned that important segments were loyal, but due to existing contracts and cost-of-switching issues — externals that dramatically changed the story.

One way to challenge your conclusions is to seek contradictions.  Look at how specific key attitudes and patterns vary by subgroup.  Examine differences by gender, age and geography, or if it’s a B2B study, by job function, industry and company size. Does the story still hold?  Sometimes the obvious is masking something that might require a bit more intellectual rigor.  As researchers, it’s our job to be disciplined and get beyond face value.  What’s the story behind the story?

Validate your results

We have to validate our data, especially when the research conclusions seem too-good-to-be-true. Whenever possible, validate by comparing with other data sources. You want to determine if relevant external data points also show consistencies about the customer attitudes and behaviors you’re examining in this particular study. A couple of examples:

• In a recent survey project, the client used social media monitoring to find customer stories and anecdotes that reflected behaviors measured in the survey project. Not the most rigorous methodology, perhaps, but it did provide real-world examples.
• In many studies, I have had clients doing online surveys conduct a small number of follow-up IDIs with respondents to access some of the “why” context.

Market Trend or Coincidence?

Sometimes as researchers we get so excited about our data that we jump.  But remember, a few coincidences don’t justify a conclusion.  We have to make sure that the story we’re sharing can be confirmed, replicated, and demonstrated with solid proof points. Otherwise we must present those conclusions as hypotheses, or “directional”. That’s not necessarily a bad thing, but it is different, and we have to have the discipline to recognize the difference. Otherwise we’re just peddling bad shoes.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Do you cringe whenever you see a survey design crafted by one of your colleagues?

Do you question whether your choice of scales and answer options is yielding the best data?

Has your organization ever released an online survey—only to get results that are too vague to be actionable?

Let’s be honest—a lot of people with no training write surveys these days.  And it shows.  If you or your colleagues are in New York please join me for a crash course in online survey design.

All students will receive a workbook and online access to additional materials.

Our agenda:

• 5 steps to planning a successful online survey
• How to avoid common survey design problems
• All about question types and scales
• How to minimize bias and maximize honesty
• Tips for getting people to complete your questionnaire

This event is being hosted by Research Rockstar, the NY AMA, and Greenbook.

Learn more | Register today

What does your target market think of your brand?

How does your target market perceive your brand as compared to your competitors’ brands?

While brand research can be a very complex, exhaustive exercise, in many cases a simple approach may suffice. 

If you plan to do your own brand perception research using online surveys, here are some tips.

How your brand is perceived

For brand perceptions, a quick and easy way to collect data is to ask, “Which of the following words would you use to describe our company?” Then give them a list of varying words and allow them to pick up to three. It’s a simple format for the respondents, and gives you very useful insights.  Do people think of your brand as “smart” and “fun” or “stable” and “safe”? Are your competitors perceived as “friendly” and “creative” or “slow” and “boring”?

Other perceptions that we commonly seek to measure in research:

• This is a company that values its customers
• This is an innovative company
• This is a company that offers products or services that are a good value (or a good value for the dollar)

These types of brand questions are going to vary by product category and target market. B2B companies will have very different questions than B2C, and so on.

Brand Perception Research, Realistically

In an ideal world, a company would do very comprehensive brand perception research. But that type of time, and budget, is not always an option. With some careful planning, many companies can learn quite a lot from a short, online survey approach.

If you’d like to receive more free Market Research tips, click HERE to sign up for Research Rockstar’s Market Research Newsletter.

With the summer season upon us, along comes a most welcomed relief for many market research project managers. The workload slows and creates a great opportunity to take care of some market research housekeeping. For survey writers, one of the best uses of slow time is to spend it creating (or updating) standard survey templates for use year-round.

Without the normal crush of deadlines, market researchers can create templates with these goals in mind:

1. Set standards.  Use this time to think carefully about how you want to standardize specific question types, along with formatting and scaling options.
2. Get approvals. Get approval from those colleagues or managers who will be involved in year-round research efforts. Explain you are constructing standard templates and want their input during the slow season—this will help everyone come crunch time.

For those newer to research, you will find that there are three common survey templates that come in handy.  A few examples of question types are included below, but there are many options—so be sure to look at several examples before you craft your templates.

Survey Template #1: Customer Satisfaction Research

A simple satisfaction survey would consist of four or five questions to gauge satisfaction and loyalty. Of course, the type of customer satisfaction and loyalty questions depends on whether you’re selling business-to-business or business-to-consumer, whether you’re selling services or products and so forth. In general, you’re going to want an overall gauge of satisfaction, which commonly uses a five-or seven-point Likert scale question. Add a few follow-up questions about satisfaction relating to the specific aspects of your products or services, as relevant, such as customer service, product’s ease of use, and possibly aesthetics.

As an example of how this might need to be modified for different product categories, let’s consider a snack product company. For this case, satisfaction measures might focus on the variety of flavors offered, response to specific flavors, and package size.

Survey Template #2: Website Feedback

If your organization interacts with customers on its website, it’s good to have a standard template for collecting website feedback. This could be used on either a transaction or a rotation basis (so that customers see it on every 10 or 20 visits), or maybe it’s something you will use once a quarter.

Common questions collect feedback on overall attractiveness, distinctiveness, and ease of use. So answer options might use a scale of “very mundane” to “very exciting”; or maybe a range from “amateurish” to “very professional”; or perhaps, “not at all easy to use” to “very easy to use”.

Other key questions may include:

• “Were you able to find the information you were looking for on our website?”
• “How likely are you to visit this web site again in the next 30 days?” (or whatever timeframe would make sense for your particular category).

Survey Template #3: Customer Service Transaction

You may want to have a survey that’s triggered every time someone completes a support call or other type of customer service transaction with your organization. This could be done through a call center, email, or even through a social media interaction such as on a Facebook fan page or via Twitter.

Typical questions ask about:

• How quickly they got a response, which gives you an objective assessment of whether or not it was timely
• Their satisfaction with the timeliness of the response
• Their satisfaction with the quality of the response

Moreover, also use it as an opportunity to make sure that the matter was completed successfully so that you can create a red flag if necessary

Market Research Planning

In my town, the Department of Public Works knows that we’re going to get a lot of snow each winter, and you can be sure that for the couple of months before winter comes, they’re stockpiling salt and sand for the roads. They’re using that relatively slow period before crunch time hits to make sure they’re prepared.

Well, it’s the same thing for survey writing. Let’s take advantage of this slow time to make sure that we have everything we’re going to need before the next crunch time hits — as it inevitably does.

Click here to check out Research Rockstar’s full line of Online Market Research Training Classes.

We all know what the most common survey design mistakes are: having surveys that are too long, too onerous, or that have questions that are leading or biased in some way.  But what about the next most common survey design errors?

1.   Too many open-ended questions.  Very often, when we’re writing a questionnaire, we realize that there are opportunities to discover new things or we are unsure of which answer options to offer. Our solution? Open-ended questions. We might ask people about unaided brand awareness; that is, “When you think of Product Category X, what brands come to mind?”  Or you might ask, “What else can our company do to improve your satisfaction with our services or products?”  Or you might have a question followed by a list of possible answers, including an “Other. Please specify:                      ” option. The first two examples are open-ended questions, but even the third one is expecting a lot.  Having two or three questions that require real writing is fine.  But if you ask too many, it becomes a turn-off. It is simply too onerous and few respondents will type that much.  The result? You end up with a lot of missing data.  So choose wisely, and use open-ended questions judiciously. 

2.   Excessive jargon.  If you’re doing a survey project, chances are that you have a great deal of expertise in a particular product category, industry, or topic area. And by virtue of the fact that you’re an expert, you have developed a specific language for talking about relevant issues. It’s very easy for those of us who develop areas of expertise to forget that other people simply don’t use the same language to discuss the same topics.  We have to be vigilant when we’re creating surveys to use friendly language.  Go for the lowest common denominator in terms of who’s going to be taking your survey—and use language that they are likely to use.  Excessive jargon turns people off and leads to dropouts, or worse. If they don’t really know what a term means they might guess, and you might be getting inaccurate data in return. 

3.   Forgetting your manners. It sounds trite but it is really true.  We need to be respectful of the people who are taking our surveys.  An occasional “please” and “thank you” goes a long way.  In the survey opening, use polite text to set the context and invite them to the survey.  Remember, they’re doing us a favor.  At the end of the survey, there should be a clear and distinct thank you message, especially if this is a survey going to your own customers.  I’m stunned at how abrupt many surveys end.  If I’m a customer and I’ve just given you 5, 10, or maybe even 15 minutes of my time to answer your survey, and it simply ends at the last screen, that’s not really very nice. 

Here’s some possible text:  “Thank you. Your opinions are very important and will help us to improve our products and services.”  Or, “Thank you. Your input has been extremely valuable.  Stay tuned to our company newsletter to hear how we’ll be applying these important research results.”  Let them know that it wasn’t just an academic exercise; that you plan to actually use the research. 

While it is great to see that there are so many free and low‑cost survey tools available today, such as Ask Your Target Market, SurveyGizmo, QuestionPro, SurveyMonkey, and Zoomerang, there are lots of mistakes that people can easily make when writing surveys.  Finding a great tool may be easy these days, but writing a great survey is not. 

Planning to write a questionnaire? Let Research Rockstar show you how to manage the process and avoid common mistakes. Click here for details.

If you are tracking customer satisfaction at regular intervals, say quarterly or monthly, you may have found that your colleagues want explanations for every increase or decrease in scores—even minor ones. Do the latest results show slight customer satisfaction improvement? If so, they want to know why. If the latest results show a down trend, they want to know why.

In some organizations, I find that people are quick to congratulate themselves on improvements, but willing to dismiss declines as possible “blips.” In other organizations, the culture seems to predisposition people to just the opposite: caution regarding positive news, and anguish to bad.

If you are new to managing such projects, here are some ways to handle those prone to such extremes:

• Remind them that you are tracking a trend. Especially during the first few measurement periods, we have to be cautious about drawing any hard conclusions. It may take a few measurements before you know what kind of “blip” is noise, versus a true increase or decrease.
• Be sure you are aware of contextual phenomenon. Minor fluctuations are often found to be due to things such as awareness of recent stock price performance changes, temporary events (recent marketing campaign halo effect), competitor news, and organizational changes. In some organizations, satisfaction scores can even be seasonal!
• Offer follow-up interviews. In-depth interviews (IDIs) with a subset of survey participants can be a great way to explore hypotheses you and your colleagues may have about certain results.

Customer Satisfaction Surveys That Don’t Satisfy

Ultimately, if you find it hard to manage how colleagues interpret customer satisfaction research results, it may be a clue that the survey design needs improvement. Does it include one or two open-ended questions to capture unscripted customer feedback? Does it capture specific types of customer experiences so you can see how they predict satisfaction levels? Are you capturing both satisfaction attitudes and loyalty behaviors?

We know that customer satisfaction is important, but we also know there is no one-size-fits-all approach.   Different researchers approach it differently which is appropriate—companies in different industries, with different types of client bases, do need different approaches.

Is That a Blip In Our Data, Or Are They Really Happy to See Us?

If you are new to measuring customer satisfaction, it is important to design the survey with an eye towards what types of data your internal audience will find most useful—and to be prepared to address the inevitable questions about upward or downward shifts. Whenever possible, do work with a market research professional experienced in measuring satisfaction in different industries—they will be able to advise you on how to design the survey and interpret the results.

Planning to hire a market research agency? Check out our online class on how to do it.

Let’s get a few facts on the table, shall we?

1. We all have non-researcher friends, colleagues and acquaintances who are writing their own online surveys using one of the many free or low-cost tools.
2. Despite all of the well-publicized voices shouting, “The survey is dead!”, “DIY” survey volume continues to surge.
3. We may tire of hearing, “The survey is dead!”, but we all know that the role of the survey isn’t what it used to be, and we really aren’t that sad about it. Indeed, many of us now spend more time on non-survey methods, either to develop best practices or for “live” projects.
4. Short surveys are better surveys.
5. New tools and technologies create possibilities for survey design and data analysis that never existed before.

So, what does this hodgepodge of facts mean?

The role of the survey is changing. Who creates them, how they are designed, when they are used, and their role relative to other methods is all shifting, fast. To remain effective in this new reality, survey research best practices must change, and the time is now.

New Rules of Survey Design

Some of the New Rules are driven by length, others by what is feasible given recent advances.

Then: Three to five screening questions were acceptable.

Now : One or two. Better yet, none—just collect the profiling data from your MROC database or panel provider.

Then: 15 to 20 minutes average duration was the old “short” compared to the 40 to 60 minute monsters of the old days.

Now: Ten minutes or less.

Then: Multiple questions per screen to minimize number of total screens.

Now: Shorter surveys make it worthwhile to exploit the visual appeal of one question per screen.

Then: Two to three open-ended questions at most, due to both respondent compliance and analysis headaches.

Now: Wildcard. We still want to avoid demanding too much “work” from our respondents, but these questions have become much easier to analyze. Whenever an open-ended question could replace a question with a long list of answer options, it’s worth a look.

Then: Survey design and programming were done by a professional researcher, going through multiple iterations, often taking days if not weeks.

Now: Basic survey design can be easily done by any motivated professional with the help of training, survey templates, or other support.

Then: Surveys designed with the assumption that they are the primary data source for an overall project.

Now: Surveys designed with the context that the survey is only one source of data, and is likely to be used in combination with multiple data sources.

Then: Surveys designed with the assumption that there would be one mode of data collection (typically phone, online or paper).

Now: Surveys designed for multiple modes of data collection, possibly including mobile.

I am generalizing here, and there are plenty of “buts” and “exceptions.” The reality, however, is that the landscape of survey design has changed. We can’t let yesterday’s best practices hold us back today.

Bottom Line

Those of us who remember spending hours (even days) to craft what can only politely be described as visually bland surveys can be forgiven a secret twinge of envy when we see DIYers creating visually stunning surveys in under an hour. But we have to get over ourselves; new tools are in, and the old rules are out.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Once you have taken the time to craft a fantastic questionnaire, the challenge becomes: how do you get people to actually take it? Of course, having a great list (or access to an alternate sample source) is key. But so is the way in which you present your invitation. Think about the last time someone sent you a survey invitation. What questions came to your mind before you decided yay or nay? I’ll bet it was the following three questions—even if you did not articulate them this way:

1. Why should I take the time to complete a survey? You should tell them, and be earnest. Let your participants know how much you value their input. Let them know how the data will be used. Key phrases include:

• “Your input will help us to improve our services.”
• “Your feedback will help us develop new products for clients like you.”
• “At company X, we are always seeking new ways to improve customer service. And the best way to do that is to ask the experts—our customers.”

2. What’s in it for me? Offer an incentive. Yes, it helps. The promise of future products that are aligned with their needs is nice, but a bit intangible. If this is a survey of your own customers, offer them a coupon or discount code. Other popular items include Amazon gift cards and drawings for tablet computers. Note that different states and countries have different laws about contests and drawings; be sure to get appropriate legal advice.

3. Is it going to be annoying? Let them know up front that you have a nice, short, respectful survey (and be sure that your survey is just that).

Of course, the invitation text is only part of the puzzle. For more tips on maximizing response rates, see Jeffrey Henning’s excellent article from last March, which cites the work of Weimiao Fan and Zheng Yan, and another one from May citing the work of Joel David Bloom, Ph.D., from the University at Albany .

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Are you or your colleagues interested in creating great online surveys? Have you been meaning to take a class, but perhaps found other options too boring, lengthy or costly?

Then please join me for a practical, fast class on June 9th, in Waltham MA. In just under three hours, attendees will learn the basics of planning, designing and writing great online surveys.

This is a small class format, so everyone will be able to ask questions and get answers. All attendees will receive a class workbook and sample survey designs.

Click below for details!

Research Rockstar Survey Design Class June 9

We get a lot of questions from people evaluating low-cost online survey platforms. And given that there are now more than 50 options available (see partial list at the end of this article), it is a bit confusing. Before you start evaluating options, consider these checklist items:

• Do you want a platform that integrates with a third party for panel access, or do you plan to provide your own sample?
• Do you need support for constant sum, rating, and semantic differential scales?
• Do you want to be able to impose complex quota rules?
• Do you want to be able to set up a library of commonly used questions or survey designs?
• Do you prefer simple questionnaire designs (perhaps for colleagues who may find more advanced options confusing)?
• Do you plan to do data analysis in the platform itself, or will you be exporting to SPSS or other programs?
• Do you want to customize and automate delivery of reports to colleagues?

Depending on your answers, your platform needs will vary. In fact, a “yes” answer to 2 or 3 of these will narrow your field of options down considerably.

Here are some of your lower-cost online survey platform options:

AYTM – http:/askyourtargetmarket.com/

Checkbox – http:/www.checkbox.com/

Cvent – http://www.cvent.com/

QuestionPro – http:/www.questionpro.com/

SurveyGizmo – http://www.surveygizmo.com/

SurveyMonkey – http:/www.surveymonkey.com/

Wufoo – http:/wufoo.com/ (just acquired by SurveyMonkey)

Zoomerang – http:/www.zoomerang.com/

And many, many more!

[If you haven’t done so already, click HERE to sign up for Research Rockstar’s Market Research Newsletter.]

Even if a market research project produces a pile of perfect data, we still face the fundamental challenge of analysis — making sure that we’re analyzing the results comprehensively and objectively. In other words, without bias.

Let’s say you’ve done an online survey. You identified your objectives, thought carefully about sampling, and designed a great questionnaire. You monitored data collection and carefully cleaned your dataset. Even after all this painstaking work, risk still exists. You still have to analyze the data, and it’s here that unexpected errors often creep in.

WHAT WE SEE 

Human beings have a natural tendency to look for and see things that confirm our hypotheses, or that are consistent with our personal experiences. The result? It’s as though someone has applied a highlighter to the data so that the things we expect or believe jump out at us.

WHAT WE DON’T SEE 

We tend to overlook things we don’t expect or don’t believe in, and have an unconscious desire to ignore things that are inconsistent with our experience and expectations. Amid the flood of tables, charts and graphs produced by our survey, there may well be unexpected results that are more significant and useful than anything we anticipated, but because we’re not looking for them, there’s a good chance we won’t see them. If you’re a doubter, take a look at this video.

ATTRIBUTES OF A GREAT RESEARCHER

Some people are truly gifted at market research analysis. Some key signs?

• They’re able to challenge their own assumptions
• They’re willing to play devil’s advocate and challenge the way other people are looking at the data too.
• They recognize unexpected themes that sometimes appear in a data set.
• They resist the temptation to embrace the first story the data reveals; they’ll look for multiple stories so that they can determine which ones have truly compelling value.
• They can deliver bad news, such as “You know, your baby is ugly”; meaning that your favorite project, concept, or idea is getting negative results.

The ability to objectively and thoroughly look at data to see unexpected patterns is the key to being a great market research analyst. Even when a research project is perfectly designed and executed, there’s a real risk that it will fall apart at the analysis stage, wasting time, money and an opportunity to profit from hard work.

Market Research Bias Problems

In any analysis project, researchers need to sanity check for the following challenges:

1. Positive Bias: The unconscious tendency to see what we’re searching for or expecting to find.
2. Inattentional Blindness: An unconscious tendency to miss what we’re not looking for or don’t expect.
3. Happy News Bias: The inability to acknowledge that there is some “bad” news.

If these problems are at all evident, consider the following options:

1. Challenge Assumptions: If you see results that closely match your expectations or confirm what you already know a little too neatly, challenge yourself. Embrace the discipline of being objective, or hire people who can be.
2. Develop a checklist: Develop a personal checklist to help you maintain objectivity. With experience, you’ll discover where your own biases tend to surface. A checklist of those biases can help you avoid objectivity traps.
3. Hire those with the skills you lack: Someone on the analysis team has to be able to see what the others miss, and be willing to speak up about it. Hire people who see things differently, and create an environment where they are willing to be vocal about what they see.
4. Diversify Analysts: Different people have different biases and will have different interpretations of a data set. With a team, it’s possible that you’ll cancel out each others’ biases. We each have our weaknesses and blind spots, and a team approach will help to mitigate them.

Market Research Analysis: Did You Catch a Fish or a Boot?

Embarking on a marketing research project is a lot like fishing. You can pick the day, the time, the spot, the lure and the line, but you can’t control what you hook. Before you head back to the dock, take a good look in your hand and see if what you’re holding is a five-pound trout — or an old boot.

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Imagine it’s Friday evening. You’ve been scrambling all week, and you’ve decided to unwind with a nice dinner out. The fellow at the desk next to yours has been raving about this new bistro in town—the best steak, perfect wine, and dessert to die for. So you decide to treat yourself, and…

The hostess is rude, the salad limp, the wine warm and the steak unrecognizable. What are the chances you’ll be going back there again? How likely are you to take future restaurant tips from the guy who set you up? Might you even tell a few friends about your horrid experience?

Yes, this has something to do with market research. Or, to be precise, surveys. When bad surveys are circulated, the company that sent them out becomes less trusted. The “consumer” becomes an unhappy customer, and may even tell others about their bad experience—with surveys in general or with the specific company.

WRITING GREAT SURVEYS IS IMPORTANT TO EVERYONE

Anybody who’s had a bad survey experience is likely to have a tainted perception of the process, and that can come out in a number of ways:

1. Participation: They may be less likely to take surveys in the future.
2. Attitude: They might spread negative word of mouth about the company that sent the survey or about market research as a whole.
3. Skepticism: They may be skeptical the next time they see market research results.
4. Investment. If they are business professionals, they may be less supportive of their organization’s investments in market research because they just don’t trust the process.

BEING A GOOD CITIZEN

As good citizens of the market research community, we have an obligation to make sure the questionnaires our organizations are distributing are impeccable. Even those coming from the well-intentioned but usually untrained DIYers. The challenge for many managers is the huge number of colleagues now using low-cost tools for creating surveys. An opportunity, yes. But without proper training and oversight, the chance of creating a bad survey is greater than ever. Here are some low-cost options to help avoid that:

• Examples – Provide a template library of commonly used, approved questions. Demographic questions at minimum, so that your colleagues will be collecting consistent profiling information but avoiding questions that may be too intrusive, or too vague to be useful.
• References – There are some great books out there. The “Handbook of Marketing Scales” by Bearden, Netemeyer and Haws (Sage Publications, 2011) is one favorite. A little technical, but absolutely readable to anyone willing to make an effort.
• Quality Assurance – Appoint one or two people as the Survey Q&A Reviewers, and give them the responsibility (and authority) of sanity-checking any surveys before going live (especially those being sent to your valued customers). Make sure this role is publicized and endorsed by managers.
• Keep it simple – There are a lot of tools out there, and while it might seem counterintuitive, sometimes you’re better off limiting the choices. Instead of many different question types, limit it to a handful (say, multiple and single choice, and Likert scales) to keep the surveys manageable, and therefore less prone to design abuse.
• Training – Basic skills are important. Start with new employee orientation materials and train your workforce. I’m a little biased here since Research Rockstar is an online training company, but there are also others that offer seminars and webinars, including the MRA, Burke Institute and ESOMAR.

SURVEY QUALITY IS CRITICAL

Survey quality is important to those who create them, those who take them and those who depend on their results. It’s in everyone’s best interest to assure that what the public sees reflects the quality and professionalism of the market research industry. As I’ve mentioned in other blogs, having a few good policies is a great place to start. And it might just earn you a nice dinner out come Friday, too.

[Planning to write a questionnaire? Check out a preview of Research Rockstar's questionnaire design process class.]

Typically when we think about market research surveys, we think of questionnaires that have 20, 30, or even more questions. Getting qualified people to complete these questionnaires has become a serious challenge.  One alternative is the single-question poll. After all, you’re much more likely to get high response and low dropout rates if you can simply say, “Hi, we have a single question we’d like your opinion on”, rather than requesting a novel’s worth of responses.

Facebook is making polling insanely easy these days, and several polling applications are available on Facebook. Creating a single-question poll is a snap, and then you can make it available on your fan page or your personal page, or you might invite friends to take it.

Let’s say your company has a fan page with hundreds or thousands of fans. You can simply post the poll on the page. No fan page? Facebook also gives you the option to “purchase sample.”  Only want men and women from the U.S., or only interested in men from Mexico for your particular poll? No problem. While gender and country are currently the only 2 options offered, I wouldn’t be surprised if Facebook came out with more powerful select options soon.

Facebook Polls: The Good News

1. Speed. A key advantage of Facebook polls, clearly, is speed. Results may be available within an hour.
2. Numbers. With over 600,000,000 users as of early 2011 (compare that to the US population of 310,000,000), you’ve got a lot of potential respondents.
3. Cost. Having done some experimenting with Facebook polls, I can tell you it’s very affordable and perfectly appropriate for certain types of topics. Placing a poll on Facebook is free as of March 2011 (though of course that’s subject to change), and their current sample pricing is very low.

Facebook Poll Limitations

1. Limited selects. Currently, you can choose from just gender and country (though again, I am sure this will change soon given how many options they offer for selecting audience members for their ads).
2. It’s “only” Facebook. Obviously, not everybody is on Facebook. While it does represent a broad mix of ages and countries and has a pretty even gender mix, it’s likely not the best match for specific groups such as business decision makers. My experience so far is that it’s good for consumer-type topics more than business-to-business polling.  But if you’re running a business-to-business company and you’ve got a fan page, it’s worth testing.

The Challenges of Polls in General

1. One Question. It’s a single question, so you have to craft that question carefully and understand who’s responding, keeping in mind that there’s a lot you don’t know about those respondents.
2. Polls don’t represent everyone. Some skeptics would say that the people who opt into these types of polls may not be representative of the broader population. Seems to me a single-question poll has a higher probability of broad response than a longer survey, so perhaps it balances out.
3. It may raise more questions. When you look at the results of a single-question poll it can raise more questions. You don’t have the benefit of a logical branching or skip pattern to follow up on specific answers. You have little context. Imagine a scenario where we ask participants to select which of a list of 5 features is most important when buying a tablet device. Say the poll finds that one item markedly stands out. On one hand that’s great, but on the other it raises the question of “why”? And how might that have varied by customer type, etc.? Cool data, but it leaves us begging for more.
4. Limited Uses. A single question can give you directional data, maybe even help you uncover some interesting things worth further investigation. But you aren’t going to make a million-dollar decision based on such data.

A Simple Test

I did a test the other day for $15.00.  I selected ‘men and women from the U.S.’ and had 50 responses within two hours (that’s 30 cents per response). Granted I don’t know much about these people, and I asked a pretty generic question, but it was very fast and affordable.  For topics where some data is better than no data, that can be totally appropriate.

Polls Are a Viable Option for Fast, Directional Data

Given the caveats that we don’t know much about the respondents and we don’t know enough to make extrapolations, there’s nothing wrong with asking a quick question of the Facebook population. Not every project warrants a big budget or weeks of effort. And with 600,000,000 users, there is a huge sample source just clicks away.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Do you cringe when you hear the word “policies”? Most people do. After all, policies often mean bureaucracy.  But in the case of market research, clear policies will minimize the risk of data quality headaches, customer over-surveying, ethical breaches and more.

Indeed, a thoughtful, well-communicated set of policies is more critical today than ever before, with so many people conducting ad hoc or “DIY” research. Well-intentioned individuals often make mistakes that could be avoided through awareness of a simple set of company-wide market research policies. Even organizations with central market research departments find it challenging to control “rogue” research—but promoting a set of policies will help minimize the risks.

Below are examples of market research policies that will promote basic, best practices:

1. Frequency. Over-surveying can lead to customer frustration and ultimately, poor response rates. Thus, a key policy is to specify how many times a year a single customer can be invited to participate in research.  Three times? Five times? There is no “right” answer for all organizations—it varies by customer type. But a rule should be in place. In this way, employees can avoid inundating customers with volumes of survey requests.  Of course, this also requires having a mechanism in place to track this.
2. Quality. All direct communications coming from your company are indicators of your brand’s quality, and surveys are no exception.  You must ensure that a kind of “quality control” resource exists to ensure that nothing sub-par gets released.  This job includes checking grammar and questioning content and logic.  For example, one common complaint about colleagues who do ad hoc research is that they may ask too many intrusive questions (a big turn off for customers). This resource could be a person, a team, or a defined process.
3. Permissibility. The best way to prevent unsanctioned surveys is to make sure everyone knows how to request and get approval for market research projects.  Your company can specify what types of research must be done through central market research (if it has such a department) and what can be done by other functional areas.  A simple research request process should be in place so that employees can submit a standard form that can be used to trigger an assessment and approval process.  Too onerous? Then how about a simple policy stating, “Any surveys over 10 minutes in duration must be approved by the central market research (or if none exists, marketing) department– no exceptions.”
4. Methods. Company guidelines should state policies for both qualitative and quantitative methods. For example, “All online surveys must be fewer than 30 questions.” Or, “Recruiting customers for in-depth interviews must be coordinated with the VP of sales at least two weeks ahead of time.”  These are just two simple examples, but you get the idea.
5. Incentives. An incentive policy should include guidelines for types of incentives and under what circumstances they can be given out.  Inform your employees ahead of time about whether or not your company restricts cash incentives or any type of “gifts” to customers.
6. Solicitation. A strict non-solicitation policy must be in place. Selling “under the guise of research” is entirely unethical and must be avoided. Even the appearance of solicitation can lead to big problems for your company. Surveys must not be used as thinly veiled lead generation mechanisms. [Click HERE to get more tips on survey design.]
7. Confidentiality. A confidentiality policy will ensure your employees understand how to use research information responsibly and will show your clients that you value their privacy.  Obviously, it is essential that confidential information is protected, so train people on what information is confidential, how it should be stored, and how it should be treated (internally and externally). Another realm of confidentiality lies in what company information is shared in a research study.  Consider rules that will avoid unwanted leaks. For example, a policy may be that any research related to new product concepts must be approved by the VP of marketing.

Market Research Training Via Policies

While these simple policies may appear obvious to an experienced researcher, it is important to present them to all research-related colleagues. Include policies in employee orientation materials and provide reference materials for all employees who may in any way touch market research—whether it’s the DIY kind or not. Just by raising awareness that there are policies, you will be providing subtle training on best practices.

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

During the January 11^th Twitterversity, one of the topic areas was, “About Survey Projects.”

Below is an excerpt from that section. The posts are from myself (@ResearchRocks), Annie Pettit (@Conversition), Manny Flores and Jeffrey Henning.

During the June 23, AMA MRC TweetOff session with myself, Jeffrey Henning (@JHenning), and Cathy Harrison (@VirtualMR), one topic we debated was the role of anonymity in customer satisfaction surveys.

Cathy’s point, “Customer satisfaction surveys are for measuring, not intervening.”

And Jeffrey’s, “Follow up with every dissatisfied customer who takes a survey.”

To be frank, my opinion on this topic has changed in just the past year or two. Before then, I was an ardent believer that all research must by anonymous—no matter what. I felt that any direct follow-up would show research participants that their survey responses could result in unexpected communications—and even if “helpful”, this experience could still impact future willingness to participate in research.

But in the past couple of years, two things have happened:

• First, I have been working with many clients who need to show that market research is not an academic exercise. Who need to demonstrate that research can directly, immediately, have positive outcomes. Many client-side market researchers have to negotiate for budget with non-researchers, who often view such studies as nice, but not necessarily actionable. Imposing anonymity on customer feedback reduces the research’s potential for clear, measurable usefulness.
• Second, I have seen raw data from several studies where it was obvious that participants expected follow-up. Indeed, anyone who has done a customer satisfaction survey knows that open-ended questions will often return entries such as, “The last software upgrade didn’t work—can you please fix it?” or “I have called your customer service number twice and can’t reach a live human being!”  You can bet that if they take the time to type that into a survey and you don’t follow-up, the damage will be irreparable.

Anonymity in Market Research

Yes, most surveys should be anonymous. But customer satisfaction surveys are an exception. Make it clear at the beginning or end of the survey that respondents can opt out (or opt in, if you prefer) of follow-up. Provide a phone number, web site or email address that can be used for any questions about how responses will be used.  The reality is that most customers expect follow-up.

What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.

Want to learn more about customer satisfaction research? Check out the Research Rockstar class here: ClassList.

When conducting survey projects, should you survey your own customer base (people who actually by from your company), or non-customers? Or both?

After all, you may very well have a list of customers that would be convenient for you to use for your survey project. But is that a wise choice? Maybe, maybe not.

In the quest for objectivity, we often do research with a mix of customer and non-customers. At the end of the project, we compare their responses to answer questions like:

• Do our current customers perceive our brand differently than do non-customers?
• Do our current customers value our product category differently than non-customers?
• Do the two groups differ in terms of feature requirements? Price sensitivity? Other purchase behavior?

Clearly, there is a lot of value in surveying both current customers and “potential” customers.  But realistically, it can be a lot easier and faster to survey current customers. After all, you may have their contact information so emailing them a survey invitation is a snap. They know who you are, and may even like you enough to want to help out by sharing their opinions—so response rates are good. And you know they are real—no worries about the veracity of responses.

In contrast, getting access to non-customers can be expensive, time-consuming, and risky. Sure you can buy sample from a panel provider, or seek other options (partnerships with professional associations, user groups, etc that may meet your needs). But how much will it cost? How long to arrange it? What kind of response rate will you get? How are those respondents authenticated?

So, what’s a time-constrained, budget-sensitive researcher to do?

In an ideal worked, you would examine your research objective, and then decide if it is best met by surveying current customers only, or a mix. For example, if your objective is to choose which features need to be in the next release of a software product, you may very well want both groups. Current customers, to keep them happy and maximize their likelihood to upgrade to the new release when it is available. New customers, in hopes of attracting them to the product for the first time.

But if time or budget doesn’t allow reaching non-customers?

Then you need to ask yourself two questions:

1. Are my customers an acceptable proxy for non-customers? Do I know enough about both groups to understand how different they are—to have an informed opinion of the likelihood that they have similar attitudes and behaviors relevant to the study I am planning? The answers will tell you how much of a risk you are taking, so you can deliver the research with appropriate caveats for anyone who will use it.
2. Is there an acceptable fallback? Even 10 interviews with non-customers is better than nothing. Or perhaps an outside expert with relevant experience is available (try Zintro or GLG for quick access to expert networks) for an objective sanity check.

Customer insights gathered through research deliver great value. But we always need to think about what we really mean by “customer”—current customers, or potential ones? Even if time or financial constraints prohibit you from conducting research with the ideal mix, take the time to understand the risks or consider fallback options.

[Subscribe by email to get the latest blogs as they are published. Click here:  Yes, Send me Blog posts by email!]

Some of our online market research classes have been updated, and we need feedback!  We’ll be testing the content by offering in-person seminars in Massachusetts. So here’s the deal: attend a free market research seminar, and all we ask in return is that you provide feedback.

The following topics are being scheduled:

• Introduction to Market Research Project Types & Methods (2 hours)
• Introduction to Market Research Project Management (2 hours)
• Product Concept Testing (1.5 hours)
• Designing Online Questionnaires (2 hours)
• Divide & Conquer: Introduction to Market Segmentation (1.5 hours)
• Creating Your Market Research Strategic Plan (1.5 hours)

Interested? Fill out this short form and you’ll get date and location information:  Seminars.  As soon as details are confirmed, you’ll get an invitation.

Thanks!

Never skimp on budget for:

• Quality sample.  Having quality sample is critical. Avoid sources that can’t talk credibly about their recruitment tactics or list sources. You don’t want to bake a cake with spoiled eggs.
• Sufficient sample quantity. If you are purchasing outside sample, be sure you have access to enough—or at least have a fallback strategy in place in case your incidence rates are tragically low.  I have seen many otherwise brilliantly managed projects experience dramatic schedule slips because of poor sample quantity planning.
• Expert research design. Don’t sabotage a project by fielding a poorly constructed questionnaire or applying an inappropriate methodology for the given objective.  Even if you intend to do a project in-house, getting help in this area will pay for itself by avoiding wasted time and money.
• Competent data analysis. If you spend the time and money to collect beautiful data, don’t skimp on the analysis. This can be harder than it sounds, especially for projects that have had data collection problems. In these cases, you can become so exhausted with the project that by the time data analysis is due, you are just ready to be done with it. Resist the temptation or get help.  Skipping on data analysis can backfire in 2 ways. First, someone else will look at the data and find an important result that you missed. Ouch. Second, you won’t get a good return on the research investment—leaving a bad impression that may derail you the next time you try to get budget for a market research study.
• Decent reporting.  Rushing to get the research delivered is understandable—but unfortunate typos, mislabeled charts or poor writing can turn off the audience and damage the research’s credibility.  Yes, really.

If you have people with these skills in-house—great, do it in-house. If you want to augment with outside expertise for discrete tasks, there are many boutique agencies and market research freelancers available for such work. And if you want an experienced expert to run the project for you, by all means hire a market research agency. But don’t skimp—it always backfires.

[What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.]

Well, “abuse” may be a little strong…but market research results certainly do get misused. Sometime intentionally; twisting results to confirm existing opinions, or ignoring results perceived to be inconvenient. In other cases, the misuse is accidental—issues such as poorly labeled charts or unclear methodology documentation can easily lead to erroneous conclusions.

Unintentional misuse of market research is best avoided through effective quality assurance processes and well-documented (and communicated) methodologies.  Making sure report readers understand what types of conclusions are appropriate to draw from a research effort, and making sure results are reported in a precise fashion, are both key parts of managing any market research project.

The harder challenge is preventing intentional misuse.

This is truly one of the biggest challenges for market research managers—preventing clients (internal or external ones) from purposefully manipulating results for self-serving purposes. But wait, is it the market research manager’s job to keep clients honest? Yes, at minimum, to point out inappropriate use.

What’s a market research project manager to do? Here are 2 tips to prevent market research abuse:

1. Find an executive advocate for the research. Having someone from outside the research function endorse the research does two powerful things. First, it sends a message that the research has value. After all, if an executive is taking the time to digest the results and be vocal about how they plan to use it, it must be important. Second, if the executive is associated with key conclusions from the research, it becomes harder for others to manipulate the results. It doesn’t prohibit the possibility of a constructive debate about how to interpret key findings—which is always to be encouraged. It just minimizes the risk that someone will hijack the project and make inappropriately self-serving conclusions from the data.
2. Include others in the analysis process. If you know the research is at risk of being abused, make a preemptive move by inviting a team of people to participate in the analysis process. If it’s a quantitative research project, host a meeting where you share a selection of key charts and facilitate an open discussion about their implications. If it’s a qualitative study, you can invite a group to view some video highlights or to receive a preview of selected verbatim quotes; again, to facilitate a group discussion. Including 4 to 8 people in an open discussion about research results and what they mean for the organization is a great way to keep everyone honest—and has the added benefit of promoting the research.

What do you think? Do you agree? Have a different perspective? Please add your comment here, or call the blog comments line at 508.691.6004 ext 703. Thanks!

Perhaps my favorite thing about reading blogs is that I can have a dialog with the author and fellow readers. Friendly debates or spontaneous collaborations are a lot of fun.

But when comments I share that are “pending moderator review” never appear, it really annoys me.

About 10 days ago, I read an interesting article on TMCnet—a site that I like for technology-related topics. But it just so happened that this article had some important omissions. So I posted a thoughtful reply. Nothing incendiary. Nothing rude. Just a friendly sharing of information with the author and fellow readers.

It never appeared.

After a week, I emailed the editor. Still nothing.

The original article recommends NPS (Net Promoter Score) as the optimal standard for customer satisfaction with telecommunications providers. Ummm, no. So since I didn’t get to share on the TMCnet site, let me share some information here for those of you interested in measuring customer satisfaction in the telecommunications space.

• “There are many scenarios in which customers may be satisfied with certain service levels or offerings yet refrain from recommending or referring the larger offering to their friends.” Yes, this is very true.
• “…customer referrals – should be the ultimate measure of customer satisfaction and should be cultivated to the greatest extent possible.” Not necessarily.

In telecommunications, willingness to refer is not always the best metric. Having done over a hundred research studies on telecomm topics over the past 20+ years, I know that other items can be more relevant. For example, two items that are very important in the telecomm space:

1. Willingness to renew (vs. propensity to brand switch). For some service providers, lack of brand loyalty is a huge challenge. And cost of customer acquisition can be quite high. So for them, the most useful metric can be renewal intent.
2. Interest in “add-ons” (incremental features/services that would increase $/customer). Again, because the cost of customer acquisition can be high in telecomm, some service providers focus not only on retention but on extensions; how can we sell more to the existing customer base? That’s why in telecomm you often hear people talk about raising ARPU (average revenue per user). And customers’ willingness to buy more says a lot (like how well the proposed add-ons align with their interests, and how far the brand has permission to extend).

Yes, NPS is a wonderfully efficient approach to measuring customer loyalty. But it isn’t the only one. Customer satisfaction and loyalty research is not a one-size-fits all proposition. Telecomm providers need to take the time to identify the best metrics for their research to be truly useful.

[As always, please add a comment or question here, or call the Blog Requests line (508.691.6004). Thanks!]

Every market research study has two objectives.

That’s right. Two.

There’s the stated research objective. Perhaps something likes, “Determine current levels of brand awareness in 5 key market areas,” or “Test 6 potential new marketing messages for alignment with emerging customer needs.”

Then there’s the other, assumed objective: getting engaged respondent participation. This is an implicit objective that too often gets minimized. Yes, we all know we have to do everything we can to maximize response rates, but the choice of methodology is too often driven by the research objective—not the respondents’ needs.

As researchers, we talk a lot about matching the methodology to the first objective. But given low response rates and the preciousness of qualified respondents, we need to focus a lot more on matching the methodology to the audience.

An Example

A researcher I know from a software company was upset after working with a market research agency on a huge study of IT executives. They collected over a thousand responses to an online survey, but data collection was brutally slow due to low response rates. When she finally got the data, she had a lot of important items to which there were a surprising percent of neutral or even “don’t know” responses. Putting aside that this issue should have been caught during the pre-test phase of the project, this was hugely disappointing.

I looked over the screening criteria myself, just to see what the scoop was, and it was obvious that the audience they were targeting was too senior for the 25-minute, very technical, online survey. The topic was about a fairly new technology, so chances are they were interested in the topic—but the methodology choice and level of detail was wrong.

The Shadow Objective

It’s always there. The need to match the project’s methodology with the target respondents’ preferences and behaviors. Maybe you want quantitative data, but the target group gets too many similar requests as is. Maybe you want to do focus groups, but your target population works in a field where scheduling is too uncertain for them to commit to 2 hours of time. Maybe you want to do a phone survey, but your audience has a low penetration of landlines.

Bottom line

Choosing the best methodology for any research study requires considering the project’s objective and the shadow objective. The good news? These days there are so many methods and tools that can make the research experience engaging, there is no need to be constricted by the choice of survey versus focus group.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

Don’t let employees bias customer feedback results. Or worse, don’t let high-score-seeking employees bully customers into giving them inflated scores.

If you allow employees to invite customers to take customer satisfaction feedback surveys, make sure they aren’t saying things like, “The highest score is a 10. I hope I earned a 10!” You have probably experienced this yourself, perhaps at an auto dealership or retail chain. Upon completion of your transaction, the clerk give you the feedback survey instructions (perhaps advising of a phone call you will receive, or giving you a URL to use), and then says something like, “I’m hoping you rate me as “extremely helpful.”

Obviously, such behavior not only taints the data, it can also make customers uncomfortable.

So how can you determine if your customer feedback system is being abused? Here are three easy options:

• Simple analysis. For example, if you are in retail you may be able to run the statistics on individual sales people or customer service reps to see if their scores appear artificially high or simply too consistent.
• Get outside help.  Deploy some mystery shoppers on a discovery mission. What do they experience?
• Ask. In the feedback survey itself, ask respondents if anyone told them how to respond or suggested a desirable feedback score.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

While recently looking over a questionnaire for a client’s employee survey, I was surprised to see very few questions related to technology use. Sure, there were a few perfunctory items on satisfaction with the IT department. But this was a questionnaire for a large accounting firm—its employees use technology every minute of every day. Surely there are more things to measure?

If you do an annual employee survey to gather current perceptions and attitudes—great! This is an area of research that can uncover real opportunities to boost employee satisfaction and productivity, and even identify ways to save money. But do be sure to include technology-related content—especially if your employees use computers, communications and related technologies as part of their jobs.

Not sure where to start? Take an hour to brainstorm some tech-related hypotheses. Before you know it, you will have a good set of items worthy of investigation. Here are some examples that are relevant to many organizations today:

• Our employees dislike using their desk phones; they prefer to forward their calls to their mobile phones
• Our employees are not following back-up protocols consistently
• Employees need more training on new applications
• Employees want to use (insert application or web site) but the company currently blocks it
• Employees want more leniency about what they can choose to install on company-provided computers
• Many people find the current IT help desk processes confusing, so they tend to avoid it until a problem becomes critical
• Employees are running blogs that sometimes touch on business topics, but are unaware of what disclaimers and protocols they should follow

Of course, the point is not to find out what they want and instantly provide it. There are obvious reasons why, for example, certain websites are blocked. Still, keeping an open mind and hearing this feedback may suggest a need to better explain why the policy exists or perhaps even identify situations where exceptions should be made.

Just Because You Build It, Doesn’t Mean They Will Come

Companies spend a lot of money on technology in an attempt to boost employee productivity—but how many go back to make sure it has happened? I have seen cases where employee research has uncovered surprising results—like that employees aren’t even using a telephony feature assumed to be critical, or that they avoid a specific business process because the related application is too cumbersome.

Does adding technology-related content add too much?

If the new content would make your annual employee survey too lengthy or onerous, then consider your options. Perhaps a smaller percentage of employees can be asked to complete the additional questions. Or maybe the employee base can be divided and asked to take 2 different questionnaires. Perhaps the questionnaires can be separated and done 6 months apart—so that employees don’t feel bombarded. Or maybe the tech questions can be tackled through in-house focus groups, for a more qualitative and in-depth discussion.

In any case, as with all employee research, it’s critical to make sure that upon completion, employees are thanked and next steps identified. Communicating how the research is driving specific, concrete actions that will improve the workplace will go a long way to boosting employee satisfaction. In contrast, employees who share thoughtful feedback and then see no action will be less willing the next time you ask.

[Is this a topic of interest? Interested in some help adding IT-related content to your employee research program? Contact me at kkorostoff@ResearchRockstar.com]

Peter Shankman advocated for market research today. I was there. Not 10 feet away from him when he did it.

It’s great to hear a social media authority praise—even promote—the benefit of asking customers’ about their views. In this case, Peter was making the excellent point that companies need to ask customers how they want their information delivered. At today’s presentation in downtown Boston, Peter specifically advised the business folks in attendance to take the time to ask their audiences (clients, prospects, donors, whatever) how they want to receive information. He points to the fact that the way information is delivered these days is extremely fragmented. Nobody can afford to simply guess how their audience wants to get information.  He points out that simply taking the time to ask people how they want information delivered can save huge amounts of money.

Now granted, he didn’t go beyond that to advise the best ways…or potential challenges…of asking people how they want to receive information. But as someone who has done actual primary research on this for many business clients over the years, I can tack on two bits of advice.

Self-reporting can be way off, so augment with actual behavioral data. If you simply give people a list of information sources (newspapers, magazines, TV, radio, LinkedIn, YouTube, etc) and ask them which are their preferred ones, you will get a very particular picture. If you ask the same people which ones they have used in the past 48 hours, you will get a very different picture. Why? People don’t always like to admit—either to themselves or others—how few sources they use. Consider my Aunt Sally. If you asked Aunt Sally how she gets news, she would emphatically tell you that she reads The New York Times, The Economist, and watches the evening news. Well, what she really means is that she wishes she did, but in reality, most of her news comes from the radio and her Yahoo home page.

Sure, it’s still good to ask people what they want. But balance that with some actual behavioral data as well. What have you read in the past 24 hours? What websites have you visited today? Nice, concrete questions.

Understand the behaviors associated with different information sources. This can be very enlightening. For example, let’s say IT buyer Jack works for a large retail chain. He gets information about IT companies from trade magazines, business magazines, and CNET. But what does he do with the information?  Maybe with the trade magazines he tends to rip out anything of interest and throw the rest into recycling. Business magazines? Perhaps he occasionally forwards an issue to a colleague, but less than once out of every 4 issues. CNET? Turns out he loves the email feature (email a story to a friend), and forwards at least 3 CNET articles a week to his peers—internal and external. So if I wanted to get on Jack’s radar screen and optimize the chance of him sharing info, I now know what to do.

How about you?

So do you know what information sources your target clients actually use? Trust? Share? The topic of information sources is actually pretty complex, and there are lots of other cool research tricks that can be used to understand behaviors. But as Peter Shankman said today, simply finding out what sources people use is a huge step towards optimizing a communications strategy.

[Do you get Research Rockstar blog posts via RSS? If not, here's the link: http://feeds.feedburner.com/ResearchRockstar]

This is my response to a well-meaning but misinformed article published in Forrester‘s Groundswell blog. I address the 2 most obvious factual errors below.

*****

Yes, online research communities have their place.  I ardently believe that there are many organizations that can benefit from structured online communities (MROCs or ORCs, as many now call them), or even just well-run, online customer advisory boards (link). In fact, there are some markets for which I think online communities can be one of the best ways of getting honest, objective customer insights. But let’s not oversell it; if we do, we’ll only cause a lot of heartache (and wasted research dollars).

So, please, some reality checks:

“And if you worry about whether these collections of consumers function like real communities on the Internet, you shouldn’t.”  Well, I think what the author meant is that “some of them do.” But nobody thinking about funding (and we are talking about significant funding here) an online research community should assume they are guaranteed to have the same experience. Few communities have the defining bond that cancer patients have (the example cited in the Forrester blog).  Some organizations simply don’t have customer bases so eager to connect with peers for experience sharing, support, etc.  Some do—there are some brands that have very passionate customers (Apple comes to mind). So be real: does your organization have a customer base that will be self-motivated to participate? Does your product category engender “team spirit”? If so, great; online communities may be realistic for you.

The statement, “But research communities can do things no other form of research can do. You can ask follow-up questions to get clarification or more detail. You can look at the profiles of members, to put their questions in context”, is factually  incorrect. You can do that with most types of research. Doing focus groups or interviews? You can easily ask for more context or follow-up clarifications, and do so with the benefit of reading body language (follow-up is also feasible for surveys, though the response is generally not real-time). Any half-decent researcher knows to gather contextual information (either for focus groups, interviews or survey projects), and in many cases the screening process uses techniques to ensure this context is valid.  Doing online research? There are most certainly tools available that allow you to do real-time probing; two that come readily to mind are Invoke Solutions and iModerate.  Yes, online communities can do these things too—but to say “no other form of research can” is simply untrue.

Online Research Communities have a place. But whenever we gather customer insights, or insights from the broader market, we always need to first ask, “What are the objectives? What do we want to learn? What will we do with this data?”  The answers to those questions will drive the key decision about best methodology. Should the research be “blind” (the sponsor kept anonymous to avoid bias)? Should it be quantitative? Is directional insight sufficient? Should it be done in one country or many? Should it include our customers and/or competitors’ customers? Only then can you pick which tools, or combination, will be the best choice for your market research dollars.

For more reading about online research communities and their use I recommend these articles:

MROC talk blog: LINK

From Research Live and well-regarded market research thinker Tom Ewing: LINK

From Vovici, a balanced review of one happy online community client’s experience (ABC Studios): LINK

[Do you get Research Rockstar blog posts via RSS? If not, here's the link: http://feeds.feedburner.com/ResearchRockstar]

I’ve been attending the MRA’s First Outlook conference in San Diego this week. From conversations with other folks here and some of the sessions I have attended, I’ve learned some interesting things. Here are just a few highlights before I hop on the plane home.

Cool: As airlines roll-out WiFi, they will also start doing customer satisfaction surveys in-flight. Capturing customer feedback in near real-time—very cool!

Trend: As more clients do research in-house, they are becoming increasingly knowledgeable about research tasks and their time requirements. And sometimes they learn that their agency has been taking advantage of them. One example: a client who found out that a task she had previously paid thousands for was a simple SPSS feature that takes just minutes. Yikes.

Sad: Some clients are referring to their online communities as research resources, but are clearly using them for both marketing and research. In one case, the client seemed very excited about how sanctioned “leaks” of new products to the online community creates buzz. She seemed far less interested in research integrity. Sigh.

Trivia: The famous Chevy Nova Spanish language fiasco is apparently a myth. Apparently the Nova sold very well under that name. Still, ALWAYS back translate your surveys!

Practical: In designing market research across multiple cultural groups, don’t equate language with culture association. For example, most second generation Latinos speak English well, but their cultural context is still distinct. Cultural sensitivity and context cannot be based solely on primary language.

Be There Next Year!

Overall, a great conference, good people, and a worthwhile trip. Next year’s MRA conference will be in Boston (June 9-11) Mark your calendars now!

Bored with bar and pie charts? Maybe it’s time to rethink the types of graphics you are using to display key research findings.

One graphic display that makes rare appearances in market research reports is the classic Venn diagram. It’s a wonderfully intuitive way to show overlapping groups. Attributed to John Venn, the Venn diagram was first introduced in 1880.

I have seen 100s of research reports, and written many myself, that contained statements about overlapping groups, like, “Customers who buy from retail stores and those who buy from our paper catalog increasingly prefer buying from online retailers.” OK, the statement isn’t so complicated. But wouldn’t a Venn diagram have more impact?

If your audience includes people who aren’t necessarily comfortable with lots of statistics, or who just have short attention spans, Venn diagrams are a powerful and simple way to convey overlaps, and trends in such overlaps.

In this diagram, I show how 2 customer groups increasingly overlap over time. It could be done as an animation for even more impact.

For more Venn examples in template form, check out this great Slideshare file: Venn.

And for more examples of various visual displays, please download the free eBook “Makreting Research Insights: 22 Visual Displays.” The ebook is available in the members-access section. Not yet a member? Sign-up for a free membership here: SIGN UP.

[All comments welcome! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 11/6!]

People like contests. I don’t know why. Must have something to do with why so many people like gambling.

People generally don’t like surveys. And it’s getting worse. How I long for the days when 20% response rates were considered “low.”

I’ve seen some creative strategies lately, to make surveys feel less, well…, like surveys. So the next time you need a quick customer insight blast:

• Ask employees for help. Those with company-related blogs and twitter accounts can post a link to a short questionnaire, along with a contest offer.
• Send out a one-question, open-ended poll. Here is a slightly paraphrased one that I saw recently, and it was quite effective, “What one thing would you like (brand) to do differently in 2010?” The client received lots of response, and found some surprising recurring themes. It wasn’t onerous for the respondents, and the client got some great insights.
• Offer an essay contest. A good topic might leverage the upcoming new year with a topic like, “In 500 words or less, describe your kitchen in the year 2020.” Or for more of a B2B angle, “…what will your work space look like in 2020?” What a great way to get customers engaged in brainstorming about future scenarios related to your product category.
• Lead with the contest. One company offered a photo contest. Then at the site where the online submissions were made, a relevant survey invitation popped up. While the example I saw was for a photography-related business, others could do it too. I can easily imagine a snack bar company hosting a photo contest, “Show us where you eat our bars!” The photos would be submitted online, and a survey invitation might read “We’d like your opinion on some new bar flavors.” A B2B angle could be, “Send us a picture of your ugliest office chair,” with a survey invitation about office furniture.

Obviously, these tactics aren’t a fit for all types of research, especially large quantitative projects. But making research fun and easy is a good way to get fresh feedback. And to the research purists: yes, these are self-selecting tactics, but so are a lot of panels and other sample sources these days.

Get the information, make it fun, judge is wisely, and use it appropriately—just like any other market data.

[BTW, if you do a contest, always be clear; we never want to mislead people about prizes. And be sure to abide by any contest-related laws. Contest laws do vary by country.]

[All comments welcome! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 11/6!]

What makes a useful marketing consultant? I mean, a really useful one?

Someone who…

Understands your product?
Knows how customers actually use it?
Understands customers’ perceptions of its strengths? Weaknesses?
Can see past your own biases?

Hmmm…maybe the most qualified consultant is a customer?

In market research, we are trained to treat customers as research participants. If they start to go off track by pontificating, we “re-direct” them to talk about their own personal experiences and opinions. In focus groups, we have various techniques for making someone stay on point. In survey projects, we carefully word questions in hopes that participants will report their own attitudes and behaviors.

And for many projects, it does make sense to keep participants focused on themselves. After all, a research participant can’t really know why their brother-in-law uses that brand of shampoo, or how her co-worker might feel about the price of air travel.

But if we are looking at some other marketing needs—like seeking input for improving word-of-mouth, or coming up with ideas for product improvements, the best “consultant” is one who really knows your company, your product. Sure, there are some great consultants out there. But sometimes, calling on customers as consultants is the best approach.

Calling on customers as consultants also has a hidden benefit: if your topic of interest is something that people may find too personal, too hard to be honest about—asking them indirectly can be most revealing. Jack may not be willing to tell you bluntly that he thinks your product is hard to use—that might make him feel stupid. But if you ask him how your product could be improved, he will be more comfortable suggesting that “some people” will find the product easier to use if the dials were just a little larger. It’s a twist on the old ploy, “It’s not for me, but a friend wants to know…”

What do you think?

[I welcome any and all comments! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 10/23!]

Word clouds are so cool.  The first time I found Wordle, I played with it for an hour.

But to be honest, I never really thought of it as much more than a curiosity…sort of a twist on trivia. What words are most frequent on a favorite blog’s page? Wordle would show it in cool colors and an appealing mix of horizontal and vertical lay-outs.

So when I saw that Tribe Research‘s Tribal ToolKit now includes a word cloud tool, I was eager to take it for a test drive.  How could word clouds really add insight? Be more than just a toy?

CloudMaker is a word cloud tool that can draw input from web pages, Twitter feeds (though, alas, restricted to Twitter’s self-imposed 20 tweet limit), and CSV files.  The CSV files part is the important one—since this is what allows you to import any text you want…like open-ended responses from a survey you were loathe to tally the hard way.

To get started, I set up an account (you need to buy some very reasonably-priced credits). Once I had my account, my first use was to check my blog page.  Was my content well-aligned with my intended message? Well, let’s use CloudMaker to find out. Here is the result (I could have added colors but I chose to keep it simple):

(I am not sure why it is pasting in sort of fuzzy; the original image looks much better on my computer).

The process was easy. With no training, it took me 6 minutes to specify the web page from which to draw the blog entries, and set some parameters. For example, I only wanted words that appeared at least 5 times to be included in the cloud.  I also spent a couple of minutes blocking common words (“you,” “your,” “some,” “very,” etc.).

The real power will be for people with open-ended responses to analyze.  Simply create a text file (easy to export from SPSS), and upload it to CloudMaker. You can even opt to have CloudMaker include the frequency counts. I’d also like to try this for analyzing interview transcripts.

Small is Beautiful: The 1-question Survey

There is a survey option as well. Want to know what 3 words people associate with your product, service, or overall brand?  The survey tool in CloudMaker lets you create a 1 question poll that accepts up to 3 text answers (best for 1 or 2-word answers). You get a URL to forward to survey-takers, and the software has built-in security (like preventing the same computer from answering the survey more than once). One catch: write your question carefully—once you create it and start data collection, you can’t edit it. So if you want to pre-test, plan accordingly (either do the pre-test via an email version of the question, or plan to use your CloudMaker credits for a pre-test).

Overall, Tribe Research has created an easy tool for applying word clouds to real business.

[I welcome any and all comments! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 10/23.  Our last winner was Joy Levin—Congrats Joy!!]

So many articles have been written lately about the limitations of market research. I just read another on Forbes.com.

Market Research? Not perfect? Noooo….

This is an all-too-common ploy to gain readers—by trying to make something that isn’t at all new or controversial sound as if it is.

Of course market research isn’t perfect.
Of course predicting consumer behavior is complex.
Of course sample sources can cause serious headaches.
Of course qualitative research isn’t conclusive.
Of course new technology-infused market research methods are emerging that will augment or replace traditional methods.

Did anyone ever think otherwise?

Given a choice between having a disciplined approach to discovering and measuring customer attitudes and behaviors versus taking wild-ass guesses? I’ll take a market research approach 95% of the time.

Besides (and this may be a bit self-serving), with a little training, many market research risks can be mitigated, new methods applied, and sources of frustration avoided.

And as an aside: I am wary of any consultancies with black box approaches. All I can say is, buyer beware.

Now sing it with me, “Tied…to the whipping post….Tied…to the whipping post…”

[I welcome any and all comments! Every 2 weeks I randomly select a commenter to win a Rockstar Mug: PIC. Next drawing is 10/23.  This week’s winner is Joy Levin—Congrats Joy!!]

I recently had the opportunity to serve as a guest speaker for a Vovici webinar titled, “It’s Greek to Me: Multilingual Surveys.” It’s a great topic, and one that researchers gloss over at their own peril.

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]