Market Research Training from Research Rockstar

If you are tracking customer satisfaction at regular intervals, say quarterly or monthly, you may have found that your colleagues want explanations for every increase or decrease in scores—even minor ones. Do the latest results show slight customer satisfaction improvement? If so, they want to know why. If the latest results show a down trend, they want to know why.

In some organizations, I find that people are quick to congratulate themselves on improvements, but willing to dismiss declines as possible “blips.” In other organizations, the culture seems to predisposition people to just the opposite: caution regarding positive news, and anguish to bad.

If you are new to managing such projects, here are some ways to handle those prone to such extremes:

• Remind them that you are tracking a trend. Especially during the first few measurement periods, we have to be cautious about drawing any hard conclusions. It may take a few measurements before you know what kind of “blip” is noise, versus a true increase or decrease.
• Be sure you are aware of contextual phenomenon. Minor fluctuations are often found to be due to things such as awareness of recent stock price performance changes, temporary events (recent marketing campaign halo effect), competitor news, and organizational changes. In some organizations, satisfaction scores can even be seasonal!
• Offer follow-up interviews. In-depth interviews (IDIs) with a subset of survey participants can be a great way to explore hypotheses you and your colleagues may have about certain results.

Customer Satisfaction Surveys That Don’t Satisfy

Ultimately, if you find it hard to manage how colleagues interpret customer satisfaction research results, it may be a clue that the survey design needs improvement. Does it include one or two open-ended questions to capture unscripted customer feedback? Does it capture specific types of customer experiences so you can see how they predict satisfaction levels? Are you capturing both satisfaction attitudes and loyalty behaviors?

We know that customer satisfaction is important, but we also know there is no one-size-fits-all approach.   Different researchers approach it differently which is appropriate—companies in different industries, with different types of client bases, do need different approaches.

Is That a Blip In Our Data, Or Are They Really Happy to See Us?

If you are new to measuring customer satisfaction, it is important to design the survey with an eye towards what types of data your internal audience will find most useful—and to be prepared to address the inevitable questions about upward or downward shifts. Whenever possible, do work with a market research professional experienced in measuring satisfaction in different industries—they will be able to advise you on how to design the survey and interpret the results.

Planning to hire a market research agency? Check out our online class on how to do it.

Planning to hire a market research agency for a market segmentation study? Then you have probably issued an RFP and are waiting for the proposals to come in.

When they do, what should you be looking for? Here are a few practical tips about what you should see in a market segmentation proposal.  If you see these things, it tells you that this proposal is from an experienced market research firm that understands how to mitigate the risks specific to segmentation projects.

The Kickoff Meeting

Do they specify a kickoff meeting?  How do they describe it?  Do they discuss things like using it as an opportunity to generate hypotheses for the segmentation model? Will they use it as an opportunity to discuss existing models, or to show you case studies of past market segmentation models that they’ve developed?

The kickoff meeting for this type of study is an important time for everybody to come together from the client side, as well as from the agency side, to put their best thinking forward.

Anybody who forgets to mention a kickoff meeting or only mentions it in the most cursory manner is overlooking one of the most important parts of making sure that the market segmentation study goes smoothly. A kickoff meeting is a good best practice for any market research project process—and is especially critical for complex ones, such as segmentation.

Model Development

How does the proposal describe the process of developing a market segmentation model?  And how will it be delivered?  Will the agency come to you with a model at the end of the project and say, “Voila!  Here is your market segmentation model”? That approach usually does not work well.

Market segmentation is not an absolute science.  Usually by the time you’ve done an exhaustive analysis, you’ve found at least two or three viable models that are then evaluated for final selection, using some agreed upon criteria (such as fit with sales strategy, suitability for informing product roadmaps, etc.).

My preference is to have the client involved in that process. And the proposal should set that expectation by offering to involve the client, and stating that at the end of the analysis phase the agency will share the best model options and candidly discuss their relative merits and weaknesses. No model is perfect.

Any agency that just says, “We will find the best model and that’s the model we will present,” well frankly, that’s just too simplistic.

Now, is it possible that after exhaustive analysis one really attractive model does come about? Sure, but that’s never been my experience. In fact, I’ve done projects where we’ve had four or five reasonable models to choose between.

Actionability

Do they say anything in the proposal about what will make the market segmentation actionable?  For example, do they talk about how they will be sure to find customer groups that will be feasible for you to actually reach or identify? Do they lay out a framework for how their segmentation analysis will be designed to support the tactical or strategic decisions that you’re going to want to inform by using the model?

Timeline

Segmentation takes time.  It is an iterative process. If they claim that they’re going to be doing the analysis part of the project in 7 business days or less, be skeptical.

Visual Display Examples

It’s always important to get a sense of how well a company can communicate research results through visual displays.  If their proposal includes visual display examples that you find intuitive and logical, then that gives you a good idea that they’ll be able to do the same for your project. Given a choice, avoid market research firms that can’t communicate their research results clearly.  After all, when you select your segmentation model you will need to share it within your organization—an experienced agency knows that and will know how to help you successfully communicate the model to your audience.

Bottom line

Selecting an agency for a market segmentation project is a big decision. Compared to many other types of research studies, segmentation is more complex and therefore riskier.   Evaluate the proposals you’ve received carefully to weed out agencies that clearly are not experienced or are offering a too-simple-to-be-true solution.

[Any questions about hiring a market research agency for your next market segmentation study? Check out our online class here, or email your question to KKorostoff@ResearchRockstar.com or use this form to request more information: InfoRequest].

[First in a series for market researchers planning their 2011 market research budgets]

Do you have existing research suppliers? Agencies that help you with survey projects, focus groups, community management or other needs?

If so, chances are you spend $100,000 or more per year, perhaps much more, with them.

These days, it’s also likely that your annual research budget has been lowered. Or that you are expected to do more with the same budget. In either case, the name of the game is budget efficiency.

One of the easiest ways to improve budget efficiency is to simply ask your research suppliers: “What options can you offer to reduce our research costs?” Let them know you are open to creative ideas.

A true partner will be open to getting creative on your behalf. Simply by asking them, you are also signaling to them that you are open to change—perhaps new tools or processes that they may have been reluctant to offer up proactively. After all, some of their clients are resistant to change, and they may have assumed this applies as well to you.

What can you expect after you ask for options? Here are a few examples of things you will likely hear:

Reduce data collection costs by building an in-house panel (as opposed to using a 3rd party panel). If you have not done so already, and depending on your market, an in-house panel can reduce data collection costs. Yes, there is an up-front investment to create it, but depending on your population parameters and research frequency, it can pay for itself within a year.

Reduce reporting costs by eschewing slide decks. Market research agencies put a lot of time into creating slide decks (whether though PowerPoint, Keynote or other programs). Online reporting options and chart automation tools can reduce operating costs, though there can be trade-off’s in formatting and customization.

Reduce project management fees by changing to a retainer versus per-project fee structure. Market research suppliers are, after all, consultancies. The more predictable the revenue stream, the more efficiently they operate. Some agencies will offer lower fees if they have a guarantee of consistent work or more of a consultative relationship.

Of course, with change comes risk.  Your provider will likely advise you of the risks associated with the options they offer.

But, hey, it doesn’t hurt to ask, right?  Give your market research suppliers a chance to be creative on your behalf. You just may be pleasantly surprised.

[To learn more about on-site workshops and consulting from Research Rockstar, click here: Info]

Is your company doing (or buying) market research, even though you don’t have a dedicated market research department?

You are not alone. Many companies invest in market research, without an actual, dedicated department.

If you have been thinking about expanding your market research investments but still aren’t quite ready to build an in-house team, another option is to build a virtual one.

What is a virtual market research department?

A virtual market research department is different than outsourcing to a full-service agency (which is, of course, always an option).  A virtual department works for you. They act as an extension of your company. They are accountable to you, and all costs are transparent (such as the actual out of pocket costs of doing the research).

In practical terms, a virtual market research department is usually based on finding one or two key market research consultants. Experienced project managers with skills and experiences relevant to your company.  Once you have them in place, they will take on the tasks of finding qualified resources as needed for specific projects.

What are the benefits?

Incremental growth. A virtual market research department is a great interim step.  You can start with this approach, and if your needs continue to grow, transition to a more conventional, in-house function.

Flexibility. Since these are not regular employees, you have a lot more flexibility.  You get to choose how many hours a week you need, and for what types of skills.

What are the risks?

Supervision. A virtual market research team needs supervision. Especially  during the first few months while you get to know each other. Have realistic expectations about the amount of time it will take to get them up to speed on your preferences, company policies, budgeting processes, and so on.

Budget creep. Most likely, the team will be working for you on an hourly or daily rate basis. You need to have an agreed upon monthly budget or “not-to-exceed” figure to avoid misunderstandings. Of course, you also need to be reasonable; nobody can manage three different studies in 5 hours a week. And you can’t expect them to deliver fantastic research without a budget.

Building a Virtual Market Research Department in 30 Days

Yes, it can be done in 30 days. Start by documenting your goals for the function. Precisely document your required processes (for communications and decision making) as well as your market research goals (what types of information you will need, to support what types of decisions). Once you have these things documented, you can go ahead and start vetting potential virtual team members.

There are many independent market research professionals with great experience who specifically seek out these kinds of client arrangements.  Need help finding one? Check out the many market research groups on LinkedIn, or ping me at KKorostoff@ResearchRockstar and I can make some specific recommendations.

If you engage with a market research agency to do a study for your company, you get lots of benefits. Access to research experts. Learnings from their extensive experience with similar studies.  Confidence that professionals with appropriate skills and credentials are working on your behalf.

But working with any type of consulting agency requires some level of project management on your part.  There are many risks when conducting market research; know them to mitigate them.

There are several points in a market research project where you, as the client-side manager, need to be  particularly cautious. Here are two of the big ones:

1. Risk: Ignoring sample quality. If your project is one where the agency is providing sample sources, you need to know what sources and something about their quality.  Ask how many sources the agency intends to use, what they are, and their experience with each source’s quality. A professional agency will be able to give you a clear, credible response. If you get a vague, confusing response, beware.
2. Risk: Not previewing final deliverables.  Ah the big day! The research agency is sending a sharp, knowledgeable professional to present your findings. The mistake? Not looking at their presentation at least 3 or 4 business days ahead of time.  It is totally appropriate for you to have a chance to preview and approve the presentation.  Don’t do it and you risk these very real scenarios:

• The presentation includes some conclusions that your colleagues, as experts in their market, know can’t possibly be correct. There goes the entire project’s credibility, all because of 1 or 2 poor slides.
• The presentation uses language that you just know will antagonize people. I am not saying you hide bad results; I am saying that you know your company culture and know how to present “bad news” in a productive way.
• The slides contain obvious errors.  Alas, it happens. The agency was swamped with deadlines the week your report was due, and insufficient time was spent checking your slides.  I know a research director from a Fortune 500 firm who bumped into a presenter in the parking lot 30 minutes before an executive-presentation; the speaker was still creating slides in the rental car!

Want more tips? Please check out the book on Amazon, “How to Hire & Manage Market Research Agencies.” OR, access a free chapter here: FREE.

Planning to hire a market research agency to help with a project? Before you do, be sure to create a list of questions you will use to assess potential agencies for fit and qualifications. Some questions are obvious: experience with similar projects, years in business, company size, and so on. But here are two important questions that are often overlooked:

1. Do you belong to any professional associations? The answer should start with, “Yes, we are members of…”, and should finish with some combination of:

• The MRA (Marketing Research Association)
• CASRO
• ESOMAR
• AAPOR
• QRCA

If they haven’t even heard of these groups, run. Run far.

2. What risks are we likely to run into with our project? All market research studies have some level of risk. An honest, clear-thinking researcher will easily be able to tell you what risks he or she foresees, and how they plan to mitigate them.  If you hear a response like, “There are no risks,” be skeptical.  At minimum, most studies have risks in terms of sample quality, data collection timeframe, and the possibility of “bad news” from the research results. All are manageable, but they exist. And an experienced market researcher will be able to tell you how they can be mitigated.

There are hundreds of market research agencies out there. Choose wisely.

[Want more tips on hiring market research agencies? Check out the book, on Amazon:  BOOK ]

Never skimp on budget for:

• Quality sample.  Having quality sample is critical. Avoid sources that can’t talk credibly about their recruitment tactics or list sources. You don’t want to bake a cake with spoiled eggs.
• Sufficient sample quantity. If you are purchasing outside sample, be sure you have access to enough—or at least have a fallback strategy in place in case your incidence rates are tragically low.  I have seen many otherwise brilliantly managed projects experience dramatic schedule slips because of poor sample quantity planning.
• Expert research design. Don’t sabotage a project by fielding a poorly constructed questionnaire or applying an inappropriate methodology for the given objective.  Even if you intend to do a project in-house, getting help in this area will pay for itself by avoiding wasted time and money.
• Competent data analysis. If you spend the time and money to collect beautiful data, don’t skimp on the analysis. This can be harder than it sounds, especially for projects that have had data collection problems. In these cases, you can become so exhausted with the project that by the time data analysis is due, you are just ready to be done with it. Resist the temptation or get help.  Skipping on data analysis can backfire in 2 ways. First, someone else will look at the data and find an important result that you missed. Ouch. Second, you won’t get a good return on the research investment—leaving a bad impression that may derail you the next time you try to get budget for a market research study.
• Decent reporting.  Rushing to get the research delivered is understandable—but unfortunate typos, mislabeled charts or poor writing can turn off the audience and damage the research’s credibility.  Yes, really.

If you have people with these skills in-house—great, do it in-house. If you want to augment with outside expertise for discrete tasks, there are many boutique agencies and market research freelancers available for such work. And if you want an experienced expert to run the project for you, by all means hire a market research agency. But don’t skimp—it always backfires.

[What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.]

A few days ago, I posted a blog about market research agencies and their websites.

Do these websites matter? Aren’t they just electronic billboards?

I think they do matter.  A lot of people who buy and use market research visit agency websites. And frankly, they are more likely to visit an agency site than that of a professional association (such as the MRA or CASRO) or of a trade publication (such as Quirk’s or Research Magazine). So these agency websites are, intentionally or not, a key source of client education.

And at the risk of sounding preachy, I think all market research professionals have a responsibility to promote market research ethics, professional standards, and innovation. And one of the ways to do it is through our public faces.

The Method to My Madness

I selected 6 websites from the Honomichl list and applied my 4 criteria. To refresh, the criteria are:

1. Educational content.
2. Standards and ethics.
3. Friendliness.
4. Innovate thinking.

Grades of A-D were assigned based on specific criteria within each metric. As an example, an agency that has at least 20 educational white papers, videos or similar forms of content got an A on the Education metric. As another example, an agency that has no mentions of market research ethics or adherence to ethical standards got an F on that metric. And if I had to search for an item for an hour only to find it buried in press releases, that gets a D. Please note that in some cases I may have missed relevant content—but if I spent an hour seeking out content for a given metric and could not find any, it is for all purposes, a low grade.

So how did the agencies do? I selected the following agencies from the Honomichl list:

Burke

IPSOS

Maritz

Market Probe

Synovate

TNS

And to be clear, I am not looking to pick on anyone. I did this as an exercise to see how well agencies do on these metrics.  My key take-always:

• Educational. Incredibly inconsistent, though a few shining stars.
• Standards and ethics. As a group, agencies do this poorly. None had mentions of ethics or standards on their home page or other prominent pages on their web sites.  Professional association badges are hard to find, if evident at all.
• Friendliness. Again, very inconsistent.
• Promotes innovative thinking. Very inconsistent. A couple of agencies do a good job; a couple seem to neglect this angle altogether.

So How Did They Do?

The top 2 grades go to….drum roll….IPSOS and TNS.  Both have web sites that have the highest scores on the metrics applied. IPSOS has a bit more content that promoted market research innovation. But both sites do a great job on education and friendliness.

I welcome any feedback here, or by phone: 508.691.6004 ext 705. Thanks!

Websites are the virtual storefronts of market research agencies. It’s where you often get your first impression of a potential research supplier. You peek in the window and see if the goods are narrow or diverse, dated or current. You get an impression about personality and culture. You may even get a sense of trustworthiness.

But if you have ever looked at more than 2 or 3 agency web sites, you will see that the quality and content vary dramatically. Almost dismayingly so.

So how to compare them? How to decide if an agency is worth further consideration?

I recommend using the following 4 criteria to assess market research agency websites:

Educational content. Number of white papers, videos, blogs and other content that educates client-side visitors. An agency that invests in client education is an agency that values its client relationships

Standards and ethics. Visible statements about market research ethics, and adherence to standards promoted by the industry’s professional associations such as CASRO, the MRA and ESOMAR. Choose agencies that have solid credentials; alas, there are many that do not.

Friendliness. Based on a site’s overall readability and ease of navigation. Why hire an agency that can’t communicate about research in an accessible way?

Innovative thinking. Has staff-authored books or papers on innovative topics, promotes new methods on website or blog. Perhaps even shares agency-funded “research on research.” An agency that spends some time and money investing in research methods will be able to best advise you on what methods will best meet your needs; you don’t want to go to a doctor that hasn’t been trained in 20 years, do you?

Obviously, a market research agency’s website is only one way to become exposed to its capabilities. Still, it is an important one. To find links to market research agencies and their websites, try the AMA’s Green Book, Quirk’s directory, the MRB Directory, or for a more global listing, ESOMAR’s online directory.

[This is part 1 of a 3-part blog series on market research agency web-sites: the next post will cover some of my favorite agency websites which meet the criteria above. Be sure to get the RSS feed so you don't miss parts 2 and 3: subscribe.]

When selecting a market research agency to carry out a project, many people buy on price. Often, the proposals from different agencies show similar methods, sample sources, and timelines. So of course you pick the least expensive.

Using price as the selection basis is simple and comfortable.

So here’s the catch: if you are sending RFPs only to agencies that come back to you with similar ideas, it’s time to rethink your short-list.

These days, there are innovators out there. Agencies using cooler tools, applying newer sample quality processes, and even offering new deliverables. Their methods may push you out of your comfort zone. Their proposals may be harder to read because they won’t be full of the same boilerplate you’ve been seeing for years.

But if you want research that will have an impact, break you out of a market research rut, and create more excitement among your internal clients at the end of the day, it’s time. Create a new agency short-list that includes innovative firms, perhaps even a few that you are skeptical about. You may be pleasantly surprised at the options they offer you.

[If you’d like help rethinking your agency short-list, I can help. Contact me at 508.691.6004 ext 705, or KKorostoff@ResearchRockstar.com]

[Want more market research ideas? Subscribe to the newsletter: FREENEWSLETTER]

Do you trust your market research agency?

• When you ask questions, do they listen and respond respectfully and credibly?
• When you ask for recommendations, are the responses thoughtful?
• Do they come to you proactively with suggestions and ideas?
• When you read their reports and memos, do you feel the information is reliable and credible?

I recently spoke with a long-time client who confessed that the only reason he hasn’t fired his market research agency is fear of change. For him, it’s the “devil you know…” syndrome. But because he doesn’t trust his agency, he is spending an inordinate amount of time checking their work and documenting communications via memos.  I’m working on a new short-list for him so that at least he can see what his agency options are. Will he take the plunge? I’m not sure. But I feel compelled to encourage him to look.

There are hundreds of market research agencies and hundreds of market research consultants. Changing agencies can be painful, but you deserve an agency that you can trust with your valuable market research investments.

Need to rethink your agency choices? Interested in finding agencies that offer more methodology options? Check out the directories at the MRA (http://www.bluebook.org/index.cfm) and Quirk’s (http://www.quirks.com/directory/index.aspx). Want help creating a new short-list of potential providers? Contact me at KKorostoff@ResearchRockstar.com. Thanks!

Are you bringing more market research in-house? Relying less on outside market research agencies? That can be a perfectly reasonable choice—for many reasons. But before you cut yourself off from your research agency partners, be sure to have some realistic estimates for the time it will take you to do these tasks in-house.

The biggest miscalculation people make when bringing research in-house is underestimating how much staff time it takes to analyze and report research results.  Even for a simple quantitative project—one without any multivariate analysis or modeling—you can easily spend 20-40 hours doing the data analysis, quality checks, creating visual displays, and preparing deliverables. And again, that’s a low time estimate.

Even if you have great people on staff with the right skills to do the analysis and reporting, you will be setting them up for failure by underestimating the time they need for such tasks. And we all know the brutal truth about primary market research; no matter how important and groundbreaking your insights are, it won’t matter if your audience doesn’t get them delivered in a clear, compelling way.

So what are your options?

• As you bring more research in-house, be sure to have a realistic resource plan in place. Create a best and worst case scenario for staff requirements given planned research levels.
• Establish a pool of qualified market research freelancers/consultants who can augment your staff during crunch times.
• Provide some basic training to your internal clients on what to expect from market research projects (ok, that is a bit self-serving since I do this type of training—but I still think the point is valid).
• Consider options for how you can best optimize the mix of in-house versus outsourced market research. For your organization, it may make sense to bring more research in-house—but choosing how to strike the right balance given your internal resources will have a big impact on this transition’s success.

[As always, I welcome any questions or comments. Please leave them here, email me at kkorostoff@ResearchRockstar.com, or by phone 508.691.6004 ext 705]

Seth Godin’s blog post yesterday was about a topic I have been thinking about a lot lately. It’s inspired by the aphorism, “…to a person with a hammer, every problem looks like a nail.”

Some of the pain that market research is going through right now is precisely because of this. In the field of market research, we have hammers we are very comfortable with:

Full-service research offerings. Bang.

Quantitative research. Wham.

Focus Groups. Smack.

In-depth interviews. Bam.

The challenge is that the nails have morphed more quickly than our hammers. Now we have an increasing base of customers who don’t necessarily need full-service offerings. Or whose “full-service” needs are very different than what they used to be. Still, they get whacked with the same hammer.

I also see an increasing number of nails…I mean, customer needs…that can be met with excellence using newer tools, but many long-time research suppliers are still stuck on conventional methods. I just met a client this week who specifically shared such an experience, and how it has resulted in an “aha” moment of, “…we need to reevaluate all of our longstanding research partners to make sure they are giving us the best options.”

So for my market research agency readers, I wonder: do you have the right hammer?

And for my friends on the client-side, the fact that we are at an inflection point in terms of market research tools and applications is exactly why I recommend using RFPs these days (boy am I going to get hate mail for this!). But truly, if you are about to engage on some important research, send an RFP to a few different agencies with a few different profiles, and see what you get back. Agencies with different sets of hammers will give you some fresh perspectives. At minimum, you will get new ideas and maybe even some education. And perhaps you will even find a proposal that truly does hit the nail on the head.

I recently had the opportunity to serve as a guest speaker for a Vovici webinar titled, “It’s Greek to Me: Multilingual Surveys.” It’s a great topic, and one that researchers gloss over at their own peril.

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]

So you’re planning to develop a new product, and want to know which features will be most important to potential buyers. And maybe which features could be nice-to-have, but not critical. Or maybe you want to estimate how adding a specific attribute could change potential market adoption.

These are obviously important questions. So, how to get the answers?

In many product categories, the best choice is to conduct primary market research, to get direct feedback from people in your target market. In some cases, qualitative feedback is fine—depending on your budget, analysis needs, and so on. But more commonly, in order to make firm decisions about product design, quantitative research is the best choice. If you want reliable conclusions about the priority ranking, for example, of 10 potential product features, you will want hard data.

[Do exceptions exist? Yes. There are some product categories and contexts in which primary market research is unlikely to yield reliable results. If you are wondering if you might be in that kind of situation, call me and I’ll be happy to discuss it with you.]

If you are thinking about using market research to inform product design decisions, you may be sending out an RFP to some market research agencies. And when their proposals come back to you, you will likely start hearing about data analysis techniques such as conjoint analysis (or discrete choice, which is a type of conjoint) and MaxDiff. You may get different recommendations from different market research agencies about which will be best—and that can get confusing.

In fact, one question I have heard many times from people in these situations is, “what is the difference between MaxDiff and Conjoint?” I was speaking recently with Brett Jarvis, a real expert on this topic from Sawtooth Technologies Consulting group, and he offered to write an article on the topic. Don’t panic: it’s not an article for stats geeks. It’s very friendly and includes great examples. The full article is being released in the September Research Rockstar newsletter, which will be sent out Monday September 21. So if you are not currently a newsletter subscriber, please sign up for free at [SIGN UP] to make sure you get this important article.

In the mean time, here is an excerpt from Brett’s piece:

“The reasons some people might get confused between conjoint and MaxDiff are two-fold. The first reason is that they both involve trade-offs to some extent. The respondent is effectively told that they can’t have everything and is forced to make choices. However, in a MaxDiff study the respondent evaluates a single list of items, whereas in conjoint the respondent evaluates complete products made up of various features. This brings us to the second reason. Both techniques can tell you how customers value different features. However, if you are focusing on a single list of items only, conjoint is likely more complex than is needed, whereas if you want to understand customer preferences across features, conjoint is essential.”

After you read this article, you will feel a lot more comfortable reading proposals from market research agencies that recommend these techniques.

And remember, no matter what techniques you are considering, always keep your research participants in mind. Some research designs can lead to longer, more cognitively challenging questionnaire designs—will your target audience be ok with that? Or will they balk at any surveys that take over 10 minutes? Sometimes a research design can be ideal from an analysis point of view, but if your survey takers won’t comply, a simpler approach will be a better choice.

Be sure to get the full article by signing up for our free newsletter here.

[Questions from readers of, "How to Hire & Manage Market Research Agencies."]

Q:   Can I share data collected from a previous project with a new agency?

A: Check the contract from your previous supplier, but the answer is usually, “yes.” When you hire a market research agency to do custom research for you, you typically own the data. That said, I have seen a few exceptions. Exceptions are most likely to exist if your data collection was part of an omnibus program or pre-existing data set. BTW, it’s always a good idea in your contracts to have the raw database listed as one of the required deliverables.

Q: Sometimes I get a little intimidated by the proposals I get. Some agencies send very technical details that I am not qualified to judge. What should I do?

A: First, this says more about their style, than about your competence. If this is an agency that is really into technical details, and does not appear interested in making them accessible to a non-technical client—will you really enjoy working with them? Sure, they may have great qualifications, but there is a lot to be said for style.   A great agency has the technical qualifications and the ability to communicate effectively.  Second, if you find yourself really needing to sanity-check a technical proposal, you can hire an expert do read the proposal and give an opinion. Where to find such experts? Email me with the specific nature of the proposal, and I will recommend a qualified consultant.

Q: Every time we hire a new market research agency, it starts great. But then things seem to fall apart. Status updates become infrequent, phone calls go unreturned, details slip through cracks. How do I keep the initial momentum going?

A: Most of my tips on this topic are in Chapter 8 of the book. Not enough? Here are 2 more suggestions: write up some milestone requirements in the contract.  This might include a minimum number of written status memos, participation in status calls by senior project mangers, or even completion of on-site milestone meetings. You might also consider a creative exit clause; in the contract, an exit clause is a condition under which both parties agree to end the relationship.

Another option is to become the squeaky wheel. Market research agencies are consultancies; they have multiple clients they are juggling. So if they are under-staffed at a given moment, they will have a hard time keeping up. Being a friendly but loud squeaky wheel can help you get attention. Don’t over-rely on email-it’s too easy to ignore. Phone calls work best. And if they are local, invite your project manager to lunch or breakfast meetings. Keep the tone friendly as long as possible. If your agency contact becomes inadequately responsive, escalate it. Seek out a Senior Manager or VP. A polite but firm conversation will go a long way.

Q: We have never done a large study before, but are planning one. When I hire a market research agency, what kind of invoicing should I expect?

A: Let’s say it’s a $100,000 project. The most common scenario is that it would be billed in 3 equal payments; at kick-off, at start of data collection, upon completion. Terms are usually net 30 but lots of agencies have clients that require net 45. Long projects, ones stretching to 5 months or longer, may be spread across 4 payments, again based on some milestones.  If budget planning is an issue for you, you can negotiate to have invoicing align with your quarterly budget; most agencies are flexible about that sort of thing.

More questions? Email them in or call the Blog Requests Line at 508.691.6004. Thanks!

Are you planning your first Customer Satisfaction project? If so, please take a moment and check all that apply (well, mentally anyway):

– I feel confused by all of the different methodology recommendations I am getting
– I feel a bit stunned by some of the proposed budgets
– I am wondering how my organization will actually use the results when the project is done
– I am worried that I don’t know enough to anticipate likely roadblocks

If some or all of these statements apply to you, don’t worry. These are perfectly common concerns, and just show that you are thinking carefully about the realities of conducting a customer satisfaction project.

So to help you get started, I’ll address some of these items briefly.

What methodology will be best?

There is no one-size-fits-all approach. Lots of methods can be used (including various qualitative and quantitative ones).  Your best mix will be selected based on:

• How you plan to use the research results. For example, a design meant to provide inputs to executive bonus calculations will be different than one designed primarily to inform organizational performance goals. In fact, any consultant or sales person who tells you their approach is the only legitimate one is only interested in sales—not insights.
• Your customer base. Its size, profile and geographic distribution all come into play. A customer sat (and yes, that is the vernacular) project designed for a consumer goods manufacturer in a highly competitive market with millions of customers worldwide will not be the same as one for a B2B software company with a primarily US-based client base of 500.
• Your budget. Yes, I said it. Any customer sat project can be designed (or, over-designed) to the tune of hundreds of thousands or even millions of dollars. But maybe your budget is limited. If your budget is $50,000, or even $20,000, options exist. Not all research firms will be interested (some are operationally optimized for larger studies), but plenty will be. Be honest about your budget limits and you will get appropriate proposals.

How will my organization use the final results?

The most common real-world uses of customer sat data are:

• To set and track organizational performance goals. For example, a company may have as a stated goal, “80% of customers report they will buy from us again in the next 6 months.” (Of course, just because 80% say they will, doesn’t mean 80% will in reality. Still, the fact they think they will is important).
• To generate inputs for executive bonus calculations.
• To identify opportunities for innovation (sometimes based on areas of customer dissatisfaction)
• To identify which customer touch points most directly predict loyalty behaviors (such as repeat purchasing and positive word of mouth); this creates a prioritized list of areas for improvement, optimized for the most impact.

What roadblocks will I likely encounter?

Short answer: a bunch. But the single most common issue is sample source. Do you really have a list of customers you can recruit to participate? Do you have other direct access to qualified customers? If you do, great. But a surprising number of companies do not.  I have worked with some really big clients, and some have had pitifully poor customer lists.  And since this is a customer sat project, you do need access to valid, qualified customers.

Here are 2 problems I have seen many times each:

• B2B companies that realize their lists a) are out of date (more than 10% of the names/phone numbers/email addresses are incorrect) b) have bad contact info (the customer sat survey needs to be of people who use the product, but the list is comprised of purchasing agents who buy, but do not use). Result? Significant delays and budget overruns.
• Consumer companies that overestimate the feasibility of using purchased lists to reach their customers. The result? Significant budget overruns.

Want More?

If you’d like a little more of an introduction to planning your first customer sat research project, please check out this 10 minute video on YouTube: Research Rockstar on YouTube.

Or, for a 53-minute, comprehensive introduction, check out this link to the Research Rockstar store: Store Link.

[Any questions or comments? Leave them here or call the blog requests line at 508.691.6004. Thanks!]

Today I read a blog that stated, “… the utility of market research is often minimal.  Many times the data is worthless even before the survey hits the field due to quickly changing business conditions, and consumers are over surveyed and fatigued by the constant bombardment of surveys online or elsewhere.” The blog is from The Armory, and is authored by Brendan Miller. I like Brendan’s posts—he has strong opinions and clearly enjoys innovative thinking. But given how many blogs and articles I have seen lately that express a similar point of view, I just have to respond.

Look, traditional market research is flawed. We all know that. It has its issues—sample quality being a biggie these days. And Brendan sums up another one nicely,  “Traditional research only captures a moment in time.” True, true, and market researchers are well aware of this (and advise clients accordingly).

But let’s not pretend social media is the elixir…the magic potion to cure all market research ills. Yes, social media as a research tool has real benefits and the innovation is exciting for suppliers and clients alike (tip of the hat on a nice piece to Fresh Networks). I am particularly interested in techniques for monitoring online conversations (nice intro by Beth Harte here, plus do include Crimson Hexagon).  But if we set unrealistic expectations about how fantastic social media is as a market research tool, we will ultimately disappoint clients, or worse (deliver misleading or egregiously flawed research).

Yes, it sounds great to make sweeping statements about social media-based research  “…like creating online customer forums can help marketers take an active and continuous listening approach.  Their insights will be timelier and therefore more relevant.” Timely? Perhaps if people happen to be talking about something you care about when you care about it. But alas, these methods also have inherent limitations and biases.

Heresy, you say?! Yes, social media-based market research has real limitations. Two of the major issues:

1.    The Popcorn effect (well, that’s what I call it anyway). When someone is particularly frustrated or particularly thrilled they “pop” onto a blog or user forum or review site and share a comment online. Many online forums suffer from these extremes, so we have to be careful. (In contrast, surveys capture a fuller spectrum of response including neutrals—which are a legitimate response and critical context in many cases).
2.    Online personas. How people talk, behave and portray themselves online is very different than how they do these things in-person. Ask anyone who has been on an online dating site and then met the individual in-person; the gap between online and in-person can be shocking.  And usually not in a good way. As just one example, in some markets, monitoring online communities would suggest an extremely rational set of buying behaviors backed up by shared reviews and deep, objective product evaluations. But in (gasp!) a focus group, a little discussion leads to people confessing to each other that the tie-breaker between brand A and brand B was based on an entirely irrational input (“I wanted my new HDTV to be sleeker than my brother’s”, the knowing nods of the other group members allowing the moderator to use the group dynamic to probe further and peel the onion on customer behaviors).

Bottom-line

The market researcher who clings to conventional surveys and focus groups like a life raft on a turbulent sea is going to drown. Those who judiciously add various social media and ethnographic-based methods along with some of the other fabulous new qualitative research tools out there will be able to navigate through the storm—and best help clients choose the methods (or mix) for their unique needs. But let’s not pretend that social media-based research is a magic cure-all; too many snake oil salesmen will only ultimately turn off clients and lead to a backlash.  And that’s not going to do anyone and good.

Has your market research budget been cut? If so, one of the programs most often impacted is Customer Satisfaction tracking. For firms accustomed to tracking on a continuous or quarterly basis, cutting back on this program is often a necessary, though unfortunate, reality.

So what to do? Here are 8 options to consider:

1.    Reduce frequency. Changing data collection frequency from 4 times a year to once a twice a year may be enough of a cost-savings.
2.    Reduce complexity. How long is the current survey instrument? Do your really use all of that data? How much of the data is nice-to-have versus must-have? Ask your market research agency for an option to dramatically simplify the questionnaire-how would that reduce their fees? After all, data collection, data analysis, and reporting fees would all lower. And it is not an either-or decision; maybe once a year you do the big daddy version, and 3 times a year you do the mini-version.
3.    Reduce deliverables. Have your deliverables been overkill anyway? I know many clients that fund large Customer Satisfaction trackers, and they get an overwhelming number of deliverables. Cross-tabs, models, online reporting tools, slide decks, reports by region, various forms of scorecards.  Do you really use all of it? Really? Maybe simply cutting back on deliverables will do the trick. Ask your market research agency to get creative.  If your budget for deliverables were to be cut in half for the year, what would they recommend? What options can they give you?
4.    Consider re-bidding the project. I know, if you have an ongoing program, switching agencies is painful. And the disruption costs can outweigh the fee savings. But it is an option. Be sure to include a couple of off-shore suppliers to compare fees.
5.    Evaluate your use of quant versus qual methods. Has your customer satisfaction research primarily been based on quantitative methods? Maybe now is a good time to consider qualitative research. Sure, there are important trade-offs. But if budgets are tight, it might be a good time to gather deeper information from a smaller number of accounts. One option: enlist some executives to do customer interviews. I’ve helped clients design such programs in the past, and the executives often have a very good experience. Many report that while taking time from their busy schedules is hard, there’s no substitute for hearing customer feedback firsthand.
6.    Go DIY. Yes, I said it. DIY. Hire a market research consultant to design a very simple questionnaire, 7 questions at most. And do the data collection yourself using Wufoo, Survey Monkey or Zoomerang. Keeping the questionnaire short is critical if you do-it-yourself. Be honest: you likely don’t have the staff time, skills or even tools do manage a large scale project in-house, let alone do the data analysis and reporting that would be necessary. But you could probably handle a mini-survey approach. It still gives you valuable input. Ideal? No. But it is an option.
7.    Leverage Social Media tools. While reducing conventional customer satisfaction tracking-or even putting it on hold-consider using social media as a way to encourage direct customer feedback. Have a company blog? Post requests for feedback (be sure to require moderation before comments appear, just n case you get inappropriate responses). Have a company Twitter feed? Tweet a request for feedback and provide an email address. Yes, analyzing volumes of open-ended responses is challenging, but there are tools for doing it if you really get that many responses. And if you only get a hundred or fewer, you can read through them.  It may be enough to give you a heads up about a new problem area before it spreads. And it is also a great way to hear customer requests you may not normally get.
8.    Leverage your Customer Advisory Council, as mentioned in my previous blog post.  Don’t have one? Ouch! Maybe now is a good time to get started.  Need help? Venator Partners and CustomerAdvisoryBoard.org are two companies I happen to know of that help with planning or improving customer advisory council programs.

Cutting back on Customer Satisfaction research budgets is hard. No doubt about it. Disrupting these trackers risks slower response to new problems, or slower recognition of new efforts’ successes. And since these programs often tie into executive compensation and bonus plans, any change to them can cause unrest. But these days, it may be a necessary evil. Luckily, with a little fresh thinking, some creative options do exist. You may even find that giving your Customer Satisfaction program a tweak results in some new insights.

Are you thinking about a market research project? If so, you may be debating whether you should hire an outside market research agency or do it in-house.

Of course, working with a market research agency is a big investment. Depending on the scope of your project, you may be looking at a $50,000, $100,000 or greater budget.  Then again, a DIY approach can also add up—perhaps far more than you expect.

In reality, the “cost” of a DIY project is not just the out of pocket expenses; it’s the time it takes to do it well. I have seen many clients decide to pursue a DIY, and then find out the hard way that they didn’t have the skills or time to design, implement and deliver results. Weeks evaporate, and they are left with a failed project.

[BTW, a tangent here...I notice that on Google, "market research company" is searched for about 10 times more than "market research agency."  Of course, they are the same thing, though market research companies do refer to themselves as "agencies."]

So how do you make this big decision: to do it yourself, or hire an agency?  Thanks to great online tools like SurveyMonkey, Wufoo and Zoomerang, DIY is certainly an option. DIY is best used when the following conditions are met:

• You are confident that you can write a questionnaire such that it will capture information objectively (and this may be the case if you are tackling a narrow topic and you really only need to ask, for example, eight to ten questions)
• You realistically have time to do the project management in-house
• You have the tools and skills in-house to clean the collected data and analyze it
• You have resources in-house that can report the findings in a way that will be credible to your internal colleagues
• You have access to a quality sample source (the people who will be qualified to take your questionnaire)
• You don’t need the research to be blind (you are willing to reveal your company as the sponsor)

If these conditions are all true, then choosing a DIY approach may be fine.  But if not, it’s time to hire a market research agency.

Here are some resources to help your with the process:

• For a practical guide, check out “How to Hire a Market Research Agency“
• For a deep look at best practices when hiring and managing an agency, check out “How to Hire & Manage Market Research Agencies.” Available on Amazon.
• For a directory of Market Research suppliers, check out Quirk‘s and The MRA’s Blue Book Directory.
• For examples of Market Research RFPs, to help you craft your own, check out the free samples in the Research Rockstar store (they are free, just add them to your shopping cart and you’ll get a download link).
• For examples of DIY research options check out: “Rise above the Noise: 3 DIY Exercises.”

Any questions or comments? Please leave them here or call our blog requests line at 508.691.6004. Thanks!

Market research is a useful tool, and when properly applied can yield business-improving results. But of course, it is a big investment: taking the time and money to do it right. So how can you make that go/no go decision with confidence? Complete these 4 easy tasks and you’ll be in good shape.

1. Write down your “Then what” statements.  For example, are you thinking about a project to gauge potential interest in a new product? OK, so imagine it is the end of that project: then what will you do? If the market research results come back negative, how will you use them? Will the product’s attributes be modified based on the market research results—or are they already set in stone? Will the results notably influence how the product will be marketed? Be honest: if you do this research, then what? Will your organization actually use it? Too often people just do research to confirm what they already know.  Do you really want to spend the time and money if the research is going to be ignored anyway?
2. Check for existing research. Check with your internal market research department, professional associations, and ad agency. Also check market research aggregators (like marketresearch.com, mindbranch.com and the-infoshop.com) and trade journals. Be sure you know what relevant market research already exists.  First, you may find that the data you need is already available, and for a lot less money than doing a new study from scratch.  Second, existing research might be available that provides important context.  For example, let’s say you are in the alcoholic beverage business, and you want to do some ad testing for a new vodka brand. Wouldn’t it be helpful to know if a market research firm recently published a study about your competitors’ advertising effectiveness?
3. Assess your team’s commitment. In most cases, research involves a team of at least 3 or 4 people. One person acts as the project manager (which may mean managing an outside market research agency or corralling in-house resources), usually 1 or 2 as content area experts (who may contribute to questionnaire design, etc) and usually at least 1 or 2 intended internal clients are involved (the people who will use the data at project conclusion). Now considering who is on your team, do your realistically think they will give this project their time and attention? Will they contribute to decision making? Attend milestone meetings? Be supportive when the final results come in? I have seen some potentially great market research projects fail because the person driving the project received inadequate support from internal colleagues.
4. Confirm your budget parameters.  How much budget do you have? If the project hits a roadblock, is there extra budget available? Do you have enough money to realistically hire a market research agency or do it in-house?  If you do, great. If not, you may need to just dig deeper into desk research and other  data sources.

So that’s your checklist. If you can answer these four questions, you should be in good shape to make your go/no go decision:

1. Will the market research results be used?
2. Is it true that no existing data is available that can be used instead?
3. Do you have a team that will allocate enough time to the project?
4. Do you have adequate budget?

I welcome any feedback or questions. Add them here, or call the blog requests line at 508.691.6004. Thanks!