Market Research Training from Research Rockstar

If you engage with a market research agency to do a study for your company, you get lots of benefits. Access to research experts. Learnings from their extensive experience with similar studies.  Confidence that professionals with appropriate skills and credentials are working on your behalf.

But working with any type of consulting agency requires some level of project management on your part.  There are many risks when conducting market research; know them to mitigate them.

There are several points in a market research project where you, as the client-side manager, need to be  particularly cautious. Here are two of the big ones:

1. Risk: Ignoring sample quality. If your project is one where the agency is providing sample sources, you need to know what sources and something about their quality.  Ask how many sources the agency intends to use, what they are, and their experience with each source’s quality. A professional agency will be able to give you a clear, credible response. If you get a vague, confusing response, beware.
2. Risk: Not previewing final deliverables.  Ah the big day! The research agency is sending a sharp, knowledgeable professional to present your findings. The mistake? Not looking at their presentation at least 3 or 4 business days ahead of time.  It is totally appropriate for you to have a chance to preview and approve the presentation.  Don’t do it and you risk these very real scenarios:

• The presentation includes some conclusions that your colleagues, as experts in their market, know can’t possibly be correct. There goes the entire project’s credibility, all because of 1 or 2 poor slides.
• The presentation uses language that you just know will antagonize people. I am not saying you hide bad results; I am saying that you know your company culture and know how to present “bad news” in a productive way.
• The slides contain obvious errors.  Alas, it happens. The agency was swamped with deadlines the week your report was due, and insufficient time was spent checking your slides.  I know a research director from a Fortune 500 firm who bumped into a presenter in the parking lot 30 minutes before an executive-presentation; the speaker was still creating slides in the rental car!

Want more tips? Please check out the book on Amazon, “How to Hire & Manage Market Research Agencies.” OR, access a free chapter here: FREE.

Planning to hire a market research agency to help with a project? Before you do, be sure to create a list of questions you will use to assess potential agencies for fit and qualifications. Some questions are obvious: experience with similar projects, years in business, company size, and so on. But here are two important questions that are often overlooked:

1. Do you belong to any professional associations? The answer should start with, “Yes, we are members of…”, and should finish with some combination of:

• The MRA (Marketing Research Association)
• CASRO
• ESOMAR
• AAPOR
• QRCA

If they haven’t even heard of these groups, run. Run far.

2. What risks are we likely to run into with our project? All market research studies have some level of risk. An honest, clear-thinking researcher will easily be able to tell you what risks he or she foresees, and how they plan to mitigate them.  If you hear a response like, “There are no risks,” be skeptical.  At minimum, most studies have risks in terms of sample quality, data collection timeframe, and the possibility of “bad news” from the research results. All are manageable, but they exist. And an experienced market researcher will be able to tell you how they can be mitigated.

There are hundreds of market research agencies out there. Choose wisely.

[Want more tips on hiring market research agencies? Check out the book, on Amazon:  BOOK ]

Never skimp on budget for:

• Quality sample.  Having quality sample is critical. Avoid sources that can’t talk credibly about their recruitment tactics or list sources. You don’t want to bake a cake with spoiled eggs.
• Sufficient sample quantity. If you are purchasing outside sample, be sure you have access to enough—or at least have a fallback strategy in place in case your incidence rates are tragically low.  I have seen many otherwise brilliantly managed projects experience dramatic schedule slips because of poor sample quantity planning.
• Expert research design. Don’t sabotage a project by fielding a poorly constructed questionnaire or applying an inappropriate methodology for the given objective.  Even if you intend to do a project in-house, getting help in this area will pay for itself by avoiding wasted time and money.
• Competent data analysis. If you spend the time and money to collect beautiful data, don’t skimp on the analysis. This can be harder than it sounds, especially for projects that have had data collection problems. In these cases, you can become so exhausted with the project that by the time data analysis is due, you are just ready to be done with it. Resist the temptation or get help.  Skipping on data analysis can backfire in 2 ways. First, someone else will look at the data and find an important result that you missed. Ouch. Second, you won’t get a good return on the research investment—leaving a bad impression that may derail you the next time you try to get budget for a market research study.
• Decent reporting.  Rushing to get the research delivered is understandable—but unfortunate typos, mislabeled charts or poor writing can turn off the audience and damage the research’s credibility.  Yes, really.

If you have people with these skills in-house—great, do it in-house. If you want to augment with outside expertise for discrete tasks, there are many boutique agencies and market research freelancers available for such work. And if you want an experienced expert to run the project for you, by all means hire a market research agency. But don’t skimp—it always backfires.

[What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.]

A few days ago, I posted a blog about market research agencies and their websites.

Do these websites matter? Aren’t they just electronic billboards?

I think they do matter.  A lot of people who buy and use market research visit agency websites. And frankly, they are more likely to visit an agency site than that of a professional association (such as the MRA or CASRO) or of a trade publication (such as Quirk’s or Research Magazine). So these agency websites are, intentionally or not, a key source of client education.

And at the risk of sounding preachy, I think all market research professionals have a responsibility to promote market research ethics, professional standards, and innovation. And one of the ways to do it is through our public faces.

The Method to My Madness

I selected 6 websites from the Honomichl list and applied my 4 criteria. To refresh, the criteria are:

1. Educational content.
2. Standards and ethics.
3. Friendliness.
4. Innovate thinking.

Grades of A-D were assigned based on specific criteria within each metric. As an example, an agency that has at least 20 educational white papers, videos or similar forms of content got an A on the Education metric. As another example, an agency that has no mentions of market research ethics or adherence to ethical standards got an F on that metric. And if I had to search for an item for an hour only to find it buried in press releases, that gets a D. Please note that in some cases I may have missed relevant content—but if I spent an hour seeking out content for a given metric and could not find any, it is for all purposes, a low grade.

So how did the agencies do? I selected the following agencies from the Honomichl list:

Burke

IPSOS

Maritz

Market Probe

Synovate

TNS

And to be clear, I am not looking to pick on anyone. I did this as an exercise to see how well agencies do on these metrics.  My key take-always:

• Educational. Incredibly inconsistent, though a few shining stars.
• Standards and ethics. As a group, agencies do this poorly. None had mentions of ethics or standards on their home page or other prominent pages on their web sites.  Professional association badges are hard to find, if evident at all.
• Friendliness. Again, very inconsistent.
• Promotes innovative thinking. Very inconsistent. A couple of agencies do a good job; a couple seem to neglect this angle altogether.

So How Did They Do?

The top 2 grades go to….drum roll….IPSOS and TNS.  Both have web sites that have the highest scores on the metrics applied. IPSOS has a bit more content that promoted market research innovation. But both sites do a great job on education and friendliness.

I welcome any feedback here, or by phone: 508.691.6004 ext 705. Thanks!

Websites are the virtual storefronts of market research agencies. It’s where you often get your first impression of a potential research supplier. You peek in the window and see if the goods are narrow or diverse, dated or current. You get an impression about personality and culture. You may even get a sense of trustworthiness.

But if you have ever looked at more than 2 or 3 agency web sites, you will see that the quality and content vary dramatically. Almost dismayingly so.

So how to compare them? How to decide if an agency is worth further consideration?

I recommend using the following 4 criteria to assess market research agency websites:

Educational content. Number of white papers, videos, blogs and other content that educates client-side visitors. An agency that invests in client education is an agency that values its client relationships

Standards and ethics. Visible statements about market research ethics, and adherence to standards promoted by the industry’s professional associations such as CASRO, the MRA and ESOMAR. Choose agencies that have solid credentials; alas, there are many that do not.

Friendliness. Based on a site’s overall readability and ease of navigation. Why hire an agency that can’t communicate about research in an accessible way?

Innovative thinking. Has staff-authored books or papers on innovative topics, promotes new methods on website or blog. Perhaps even shares agency-funded “research on research.” An agency that spends some time and money investing in research methods will be able to best advise you on what methods will best meet your needs; you don’t want to go to a doctor that hasn’t been trained in 20 years, do you?

Obviously, a market research agency’s website is only one way to become exposed to its capabilities. Still, it is an important one. To find links to market research agencies and their websites, try the AMA’s Green Book, Quirk’s directory, the MRB Directory, or for a more global listing, ESOMAR’s online directory.

[This is part 1 of a 3-part blog series on market research agency web-sites: the next post will cover some of my favorite agency websites which meet the criteria above. Be sure to get the RSS feed so you don't miss parts 2 and 3: subscribe.]

When selecting a market research agency to carry out a project, many people buy on price. Often, the proposals from different agencies show similar methods, sample sources, and timelines. So of course you pick the least expensive.

Using price as the selection basis is simple and comfortable.

So here’s the catch: if you are sending RFPs only to agencies that come back to you with similar ideas, it’s time to rethink your short-list.

These days, there are innovators out there. Agencies using cooler tools, applying newer sample quality processes, and even offering new deliverables. Their methods may push you out of your comfort zone. Their proposals may be harder to read because they won’t be full of the same boilerplate you’ve been seeing for years.

But if you want research that will have an impact, break you out of a market research rut, and create more excitement among your internal clients at the end of the day, it’s time. Create a new agency short-list that includes innovative firms, perhaps even a few that you are skeptical about. You may be pleasantly surprised at the options they offer you.

[If you’d like help rethinking your agency short-list, I can help. Contact me at 508.691.6004 ext 705, or KKorostoff@ResearchRockstar.com]

[Want more market research ideas? Subscribe to the newsletter: FREENEWSLETTER]

Do you trust your market research agency?

• When you ask questions, do they listen and respond respectfully and credibly?
• When you ask for recommendations, are the responses thoughtful?
• Do they come to you proactively with suggestions and ideas?
• When you read their reports and memos, do you feel the information is reliable and credible?

I recently spoke with a long-time client who confessed that the only reason he hasn’t fired his market research agency is fear of change. For him, it’s the “devil you know…” syndrome. But because he doesn’t trust his agency, he is spending an inordinate amount of time checking their work and documenting communications via memos.  I’m working on a new short-list for him so that at least he can see what his agency options are. Will he take the plunge? I’m not sure. But I feel compelled to encourage him to look.

There are hundreds of market research agencies and hundreds of market research consultants. Changing agencies can be painful, but you deserve an agency that you can trust with your valuable market research investments.

Need to rethink your agency choices? Interested in finding agencies that offer more methodology options? Check out the directories at the MRA (http://www.bluebook.org/index.cfm) and Quirk’s (http://www.quirks.com/directory/index.aspx). Want help creating a new short-list of potential providers? Contact me at KKorostoff@ResearchRockstar.com. Thanks!

Are you bringing more market research in-house? Relying less on outside market research agencies? That can be a perfectly reasonable choice—for many reasons. But before you cut yourself off from your research agency partners, be sure to have some realistic estimates for the time it will take you to do these tasks in-house.

The biggest miscalculation people make when bringing research in-house is underestimating how much staff time it takes to analyze and report research results.  Even for a simple quantitative project—one without any multivariate analysis or modeling—you can easily spend 20-40 hours doing the data analysis, quality checks, creating visual displays, and preparing deliverables. And again, that’s a low time estimate.

Even if you have great people on staff with the right skills to do the analysis and reporting, you will be setting them up for failure by underestimating the time they need for such tasks. And we all know the brutal truth about primary market research; no matter how important and groundbreaking your insights are, it won’t matter if your audience doesn’t get them delivered in a clear, compelling way.

So what are your options?

• As you bring more research in-house, be sure to have a realistic resource plan in place. Create a best and worst case scenario for staff requirements given planned research levels.
• Establish a pool of qualified market research freelancers/consultants who can augment your staff during crunch times.
• Provide some basic training to your internal clients on what to expect from market research projects (ok, that is a bit self-serving since I do this type of training—but I still think the point is valid).
• Consider options for how you can best optimize the mix of in-house versus outsourced market research. For your organization, it may make sense to bring more research in-house—but choosing how to strike the right balance given your internal resources will have a big impact on this transition’s success.

[As always, I welcome any questions or comments. Please leave them here, email me at kkorostoff@ResearchRockstar.com, or by phone 508.691.6004 ext 705]

Seth Godin’s blog post yesterday was about a topic I have been thinking about a lot lately. It’s inspired by the aphorism, “…to a person with a hammer, every problem looks like a nail.”

Some of the pain that market research is going through right now is precisely because of this. In the field of market research, we have hammers we are very comfortable with:

Full-service research offerings. Bang.

Quantitative research. Wham.

Focus Groups. Smack.

In-depth interviews. Bam.

The challenge is that the nails have morphed more quickly than our hammers. Now we have an increasing base of customers who don’t necessarily need full-service offerings. Or whose “full-service” needs are very different than what they used to be. Still, they get whacked with the same hammer.

I also see an increasing number of nails…I mean, customer needs…that can be met with excellence using newer tools, but many long-time research suppliers are still stuck on conventional methods. I just met a client this week who specifically shared such an experience, and how it has resulted in an “aha” moment of, “…we need to reevaluate all of our longstanding research partners to make sure they are giving us the best options.”

So for my market research agency readers, I wonder: do you have the right hammer?

And for my friends on the client-side, the fact that we are at an inflection point in terms of market research tools and applications is exactly why I recommend using RFPs these days (boy am I going to get hate mail for this!). But truly, if you are about to engage on some important research, send an RFP to a few different agencies with a few different profiles, and see what you get back. Agencies with different sets of hammers will give you some fresh perspectives. At minimum, you will get new ideas and maybe even some education. And perhaps you will even find a proposal that truly does hit the nail on the head.

I recently had the opportunity to serve as a guest speaker for a Vovici webinar titled, “It’s Greek to Me: Multilingual Surveys.” It’s a great topic, and one that researchers gloss over at their own peril.

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]

So you’re planning to develop a new product, and want to know which features will be most important to potential buyers. And maybe which features could be nice-to-have, but not critical. Or maybe you want to estimate how adding a specific attribute could change potential market adoption.

These are obviously important questions. So, how to get the answers?

In many product categories, the best choice is to conduct primary market research, to get direct feedback from people in your target market. In some cases, qualitative feedback is fine—depending on your budget, analysis needs, and so on. But more commonly, in order to make firm decisions about product design, quantitative research is the best choice. If you want reliable conclusions about the priority ranking, for example, of 10 potential product features, you will want hard data.

[Do exceptions exist? Yes. There are some product categories and contexts in which primary market research is unlikely to yield reliable results. If you are wondering if you might be in that kind of situation, call me and I’ll be happy to discuss it with you.]

If you are thinking about using market research to inform product design decisions, you may be sending out an RFP to some market research agencies. And when their proposals come back to you, you will likely start hearing about data analysis techniques such as conjoint analysis (or discrete choice, which is a type of conjoint) and MaxDiff. You may get different recommendations from different market research agencies about which will be best—and that can get confusing.

In fact, one question I have heard many times from people in these situations is, “what is the difference between MaxDiff and Conjoint?” I was speaking recently with Brett Jarvis, a real expert on this topic from Sawtooth Technologies Consulting group, and he offered to write an article on the topic. Don’t panic: it’s not an article for stats geeks. It’s very friendly and includes great examples. The full article is being released in the September Research Rockstar newsletter, which will be sent out Monday September 21. So if you are not currently a newsletter subscriber, please sign up for free at [SIGN UP] to make sure you get this important article.

In the mean time, here is an excerpt from Brett’s piece:

“The reasons some people might get confused between conjoint and MaxDiff are two-fold. The first reason is that they both involve trade-offs to some extent. The respondent is effectively told that they can’t have everything and is forced to make choices. However, in a MaxDiff study the respondent evaluates a single list of items, whereas in conjoint the respondent evaluates complete products made up of various features. This brings us to the second reason. Both techniques can tell you how customers value different features. However, if you are focusing on a single list of items only, conjoint is likely more complex than is needed, whereas if you want to understand customer preferences across features, conjoint is essential.”

After you read this article, you will feel a lot more comfortable reading proposals from market research agencies that recommend these techniques.

And remember, no matter what techniques you are considering, always keep your research participants in mind. Some research designs can lead to longer, more cognitively challenging questionnaire designs—will your target audience be ok with that? Or will they balk at any surveys that take over 10 minutes? Sometimes a research design can be ideal from an analysis point of view, but if your survey takers won’t comply, a simpler approach will be a better choice.

Be sure to get the full article by signing up for our free newsletter here.

[Questions from readers of, "How to Hire & Manage Market Research Agencies."]

Q:   Can I share data collected from a previous project with a new agency?

A: Check the contract from your previous supplier, but the answer is usually, “yes.” When you hire a market research agency to do custom research for you, you typically own the data. That said, I have seen a few exceptions. Exceptions are most likely to exist if your data collection was part of an omnibus program or pre-existing data set. BTW, it’s always a good idea in your contracts to have the raw database listed as one of the required deliverables.

Q: Sometimes I get a little intimidated by the proposals I get. Some agencies send very technical details that I am not qualified to judge. What should I do?

A: First, this says more about their style, than about your competence. If this is an agency that is really into technical details, and does not appear interested in making them accessible to a non-technical client—will you really enjoy working with them? Sure, they may have great qualifications, but there is a lot to be said for style.   A great agency has the technical qualifications and the ability to communicate effectively.  Second, if you find yourself really needing to sanity-check a technical proposal, you can hire an expert do read the proposal and give an opinion. Where to find such experts? Email me with the specific nature of the proposal, and I will recommend a qualified consultant.

Q: Every time we hire a new market research agency, it starts great. But then things seem to fall apart. Status updates become infrequent, phone calls go unreturned, details slip through cracks. How do I keep the initial momentum going?

A: Most of my tips on this topic are in Chapter 8 of the book. Not enough? Here are 2 more suggestions: write up some milestone requirements in the contract.  This might include a minimum number of written status memos, participation in status calls by senior project mangers, or even completion of on-site milestone meetings. You might also consider a creative exit clause; in the contract, an exit clause is a condition under which both parties agree to end the relationship.

Another option is to become the squeaky wheel. Market research agencies are consultancies; they have multiple clients they are juggling. So if they are under-staffed at a given moment, they will have a hard time keeping up. Being a friendly but loud squeaky wheel can help you get attention. Don’t over-rely on email-it’s too easy to ignore. Phone calls work best. And if they are local, invite your project manager to lunch or breakfast meetings. Keep the tone friendly as long as possible. If your agency contact becomes inadequately responsive, escalate it. Seek out a Senior Manager or VP. A polite but firm conversation will go a long way.

Q: We have never done a large study before, but are planning one. When I hire a market research agency, what kind of invoicing should I expect?

A: Let’s say it’s a $100,000 project. The most common scenario is that it would be billed in 3 equal payments; at kick-off, at start of data collection, upon completion. Terms are usually net 30 but lots of agencies have clients that require net 45. Long projects, ones stretching to 5 months or longer, may be spread across 4 payments, again based on some milestones.  If budget planning is an issue for you, you can negotiate to have invoicing align with your quarterly budget; most agencies are flexible about that sort of thing.

More questions? Email them in or call the Blog Requests Line at 508.691.6004. Thanks!

Are you planning your first Customer Satisfaction project? If so, please take a moment and check all that apply (well, mentally anyway):

– I feel confused by all of the different methodology recommendations I am getting
– I feel a bit stunned by some of the proposed budgets
– I am wondering how my organization will actually use the results when the project is done
– I am worried that I don’t know enough to anticipate likely roadblocks

If some or all of these statements apply to you, don’t worry. These are perfectly common concerns, and just show that you are thinking carefully about the realities of conducting a customer satisfaction project.

So to help you get started, I’ll address some of these items briefly.

What methodology will be best?

There is no one-size-fits-all approach. Lots of methods can be used (including various qualitative and quantitative ones).  Your best mix will be selected based on:

• How you plan to use the research results. For example, a design meant to provide inputs to executive bonus calculations will be different than one designed primarily to inform organizational performance goals. In fact, any consultant or sales person who tells you their approach is the only legitimate one is only interested in sales—not insights.
• Your customer base. Its size, profile and geographic distribution all come into play. A customer sat (and yes, that is the vernacular) project designed for a consumer goods manufacturer in a highly competitive market with millions of customers worldwide will not be the same as one for a B2B software company with a primarily US-based client base of 500.
• Your budget. Yes, I said it. Any customer sat project can be designed (or, over-designed) to the tune of hundreds of thousands or even millions of dollars. But maybe your budget is limited. If your budget is $50,000, or even $20,000, options exist. Not all research firms will be interested (some are operationally optimized for larger studies), but plenty will be. Be honest about your budget limits and you will get appropriate proposals.

How will my organization use the final results?

The most common real-world uses of customer sat data are:

• To set and track organizational performance goals. For example, a company may have as a stated goal, “80% of customers report they will buy from us again in the next 6 months.” (Of course, just because 80% say they will, doesn’t mean 80% will in reality. Still, the fact they think they will is important).
• To generate inputs for executive bonus calculations.
• To identify opportunities for innovation (sometimes based on areas of customer dissatisfaction)
• To identify which customer touch points most directly predict loyalty behaviors (such as repeat purchasing and positive word of mouth); this creates a prioritized list of areas for improvement, optimized for the most impact.

What roadblocks will I likely encounter?

Short answer: a bunch. But the single most common issue is sample source. Do you really have a list of customers you can recruit to participate? Do you have other direct access to qualified customers? If you do, great. But a surprising number of companies do not.  I have worked with some really big clients, and some have had pitifully poor customer lists.  And since this is a customer sat project, you do need access to valid, qualified customers.

Here are 2 problems I have seen many times each:

• B2B companies that realize their lists a) are out of date (more than 10% of the names/phone numbers/email addresses are incorrect) b) have bad contact info (the customer sat survey needs to be of people who use the product, but the list is comprised of purchasing agents who buy, but do not use). Result? Significant delays and budget overruns.
• Consumer companies that overestimate the feasibility of using purchased lists to reach their customers. The result? Significant budget overruns.

Want More?

If you’d like a little more of an introduction to planning your first customer sat research project, please check out this 10 minute video on YouTube: Research Rockstar on YouTube.

Or, for a 53-minute, comprehensive introduction, check out this link to the Research Rockstar store: Store Link.

[Any questions or comments? Leave them here or call the blog requests line at 508.691.6004. Thanks!]

Today I read a blog that stated, “… the utility of market research is often minimal.  Many times the data is worthless even before the survey hits the field due to quickly changing business conditions, and consumers are over surveyed and fatigued by the constant bombardment of surveys online or elsewhere.” The blog is from The Armory, and is authored by Brendan Miller. I like Brendan’s posts—he has strong opinions and clearly enjoys innovative thinking. But given how many blogs and articles I have seen lately that express a similar point of view, I just have to respond.

Look, traditional market research is flawed. We all know that. It has its issues—sample quality being a biggie these days. And Brendan sums up another one nicely,  “Traditional research only captures a moment in time.” True, true, and market researchers are well aware of this (and advise clients accordingly).

But let’s not pretend social media is the elixir…the magic potion to cure all market research ills. Yes, social media as a research tool has real benefits and the innovation is exciting for suppliers and clients alike (tip of the hat on a nice piece to Fresh Networks). I am particularly interested in techniques for monitoring online conversations (nice intro by Beth Harte here, plus do include Crimson Hexagon).  But if we set unrealistic expectations about how fantastic social media is as a market research tool, we will ultimately disappoint clients, or worse (deliver misleading or egregiously flawed research).

Yes, it sounds great to make sweeping statements about social media-based research  “…like creating online customer forums can help marketers take an active and continuous listening approach.  Their insights will be timelier and therefore more relevant.” Timely? Perhaps if people happen to be talking about something you care about when you care about it. But alas, these methods also have inherent limitations and biases.

Heresy, you say?! Yes, social media-based market research has real limitations. Two of the major issues:

1.    The Popcorn effect (well, that’s what I call it anyway). When someone is particularly frustrated or particularly thrilled they “pop” onto a blog or user forum or review site and share a comment online. Many online forums suffer from these extremes, so we have to be careful. (In contrast, surveys capture a fuller spectrum of response including neutrals—which are a legitimate response and critical context in many cases).
2.    Online personas. How people talk, behave and portray themselves online is very different than how they do these things in-person. Ask anyone who has been on an online dating site and then met the individual in-person; the gap between online and in-person can be shocking.  And usually not in a good way. As just one example, in some markets, monitoring online communities would suggest an extremely rational set of buying behaviors backed up by shared reviews and deep, objective product evaluations. But in (gasp!) a focus group, a little discussion leads to people confessing to each other that the tie-breaker between brand A and brand B was based on an entirely irrational input (“I wanted my new HDTV to be sleeker than my brother’s”, the knowing nods of the other group members allowing the moderator to use the group dynamic to probe further and peel the onion on customer behaviors).

Bottom-line

The market researcher who clings to conventional surveys and focus groups like a life raft on a turbulent sea is going to drown. Those who judiciously add various social media and ethnographic-based methods along with some of the other fabulous new qualitative research tools out there will be able to navigate through the storm—and best help clients choose the methods (or mix) for their unique needs. But let’s not pretend that social media-based research is a magic cure-all; too many snake oil salesmen will only ultimately turn off clients and lead to a backlash.  And that’s not going to do anyone and good.

Has your market research budget been cut? If so, one of the programs most often impacted is Customer Satisfaction tracking. For firms accustomed to tracking on a continuous or quarterly basis, cutting back on this program is often a necessary, though unfortunate, reality.

So what to do? Here are 8 options to consider:

1.    Reduce frequency. Changing data collection frequency from 4 times a year to once a twice a year may be enough of a cost-savings.
2.    Reduce complexity. How long is the current survey instrument? Do your really use all of that data? How much of the data is nice-to-have versus must-have? Ask your market research agency for an option to dramatically simplify the questionnaire-how would that reduce their fees? After all, data collection, data analysis, and reporting fees would all lower. And it is not an either-or decision; maybe once a year you do the big daddy version, and 3 times a year you do the mini-version.
3.    Reduce deliverables. Have your deliverables been overkill anyway? I know many clients that fund large Customer Satisfaction trackers, and they get an overwhelming number of deliverables. Cross-tabs, models, online reporting tools, slide decks, reports by region, various forms of scorecards.  Do you really use all of it? Really? Maybe simply cutting back on deliverables will do the trick. Ask your market research agency to get creative.  If your budget for deliverables were to be cut in half for the year, what would they recommend? What options can they give you?
4.    Consider re-bidding the project. I know, if you have an ongoing program, switching agencies is painful. And the disruption costs can outweigh the fee savings. But it is an option. Be sure to include a couple of off-shore suppliers to compare fees.
5.    Evaluate your use of quant versus qual methods. Has your customer satisfaction research primarily been based on quantitative methods? Maybe now is a good time to consider qualitative research. Sure, there are important trade-offs. But if budgets are tight, it might be a good time to gather deeper information from a smaller number of accounts. One option: enlist some executives to do customer interviews. I’ve helped clients design such programs in the past, and the executives often have a very good experience. Many report that while taking time from their busy schedules is hard, there’s no substitute for hearing customer feedback firsthand.
6.    Go DIY. Yes, I said it. DIY. Hire a market research consultant to design a very simple questionnaire, 7 questions at most. And do the data collection yourself using Wufoo, Survey Monkey or Zoomerang. Keeping the questionnaire short is critical if you do-it-yourself. Be honest: you likely don’t have the staff time, skills or even tools do manage a large scale project in-house, let alone do the data analysis and reporting that would be necessary. But you could probably handle a mini-survey approach. It still gives you valuable input. Ideal? No. But it is an option.
7.    Leverage Social Media tools. While reducing conventional customer satisfaction tracking-or even putting it on hold-consider using social media as a way to encourage direct customer feedback. Have a company blog? Post requests for feedback (be sure to require moderation before comments appear, just n case you get inappropriate responses). Have a company Twitter feed? Tweet a request for feedback and provide an email address. Yes, analyzing volumes of open-ended responses is challenging, but there are tools for doing it if you really get that many responses. And if you only get a hundred or fewer, you can read through them.  It may be enough to give you a heads up about a new problem area before it spreads. And it is also a great way to hear customer requests you may not normally get.
8.    Leverage your Customer Advisory Council, as mentioned in my previous blog post.  Don’t have one? Ouch! Maybe now is a good time to get started.  Need help? Venator Partners and CustomerAdvisoryBoard.org are two companies I happen to know of that help with planning or improving customer advisory council programs.

Cutting back on Customer Satisfaction research budgets is hard. No doubt about it. Disrupting these trackers risks slower response to new problems, or slower recognition of new efforts’ successes. And since these programs often tie into executive compensation and bonus plans, any change to them can cause unrest. But these days, it may be a necessary evil. Luckily, with a little fresh thinking, some creative options do exist. You may even find that giving your Customer Satisfaction program a tweak results in some new insights.

Are you thinking about a market research project? If so, you may be debating whether you should hire an outside market research agency or do it in-house.

Of course, working with a market research agency is a big investment. Depending on the scope of your project, you may be looking at a $50,000, $100,000 or greater budget.  Then again, a DIY approach can also add up—perhaps far more than you expect.

In reality, the “cost” of a DIY project is not just the out of pocket expenses; it’s the time it takes to do it well. I have seen many clients decide to pursue a DIY, and then find out the hard way that they didn’t have the skills or time to design, implement and deliver results. Weeks evaporate, and they are left with a failed project.

[BTW, a tangent here...I notice that on Google, "market research company" is searched for about 10 times more than "market research agency."  Of course, they are the same thing, though market research companies do refer to themselves as "agencies."]

So how do you make this big decision: to do it yourself, or hire an agency?  Thanks to great online tools like SurveyMonkey, Wufoo and Zoomerang, DIY is certainly an option. DIY is best used when the following conditions are met:

• You are confident that you can write a questionnaire such that it will capture information objectively (and this may be the case if you are tackling a narrow topic and you really only need to ask, for example, eight to ten questions)
• You realistically have time to do the project management in-house
• You have the tools and skills in-house to clean the collected data and analyze it
• You have resources in-house that can report the findings in a way that will be credible to your internal colleagues
• You have access to a quality sample source (the people who will be qualified to take your questionnaire)
• You don’t need the research to be blind (you are willing to reveal your company as the sponsor)

If these conditions are all true, then choosing a DIY approach may be fine.  But if not, it’s time to hire a market research agency.

Here are some resources to help your with the process:

• For a practical guide, check out “How to Hire a Market Research Agency“
• For a deep look at best practices when hiring and managing an agency, check out “How to Hire & Manage Market Research Agencies.” Available on Amazon.
• For a directory of Market Research suppliers, check out Quirk’s and The MRA’s Blue Book Directory.
• For examples of Market Research RFPs, to help you craft your own, check out the free samples in the Research Rockstar store (they are free, just add them to your shopping cart and you’ll get a download link).
• For examples of DIY research options check out: “Rise above the Noise: 3 DIY Exercises.”

Any questions or comments? Please leave them here or call our blog requests line at 508.691.6004. Thanks!

Market research is a useful tool, and when properly applied can yield business-improving results. But of course, it is a big investment: taking the time and money to do it right. So how can you make that go/no go decision with confidence? Complete these 4 easy tasks and you’ll be in good shape.

1. Write down your “Then what” statements.  For example, are you thinking about a project to gauge potential interest in a new product? OK, so imagine it is the end of that project: then what will you do? If the market research results come back negative, how will you use them? Will the product’s attributes be modified based on the market research results—or are they already set in stone? Will the results notably influence how the product will be marketed? Be honest: if you do this research, then what? Will your organization actually use it? Too often people just do research to confirm what they already know.  Do you really want to spend the time and money if the research is going to be ignored anyway?
2. Check for existing research. Check with your internal market research department, professional associations, and ad agency. Also check market research aggregators (like marketresearch.com, mindbranch.com and the-infoshop.com) and trade journals. Be sure you know what relevant market research already exists.  First, you may find that the data you need is already available, and for a lot less money than doing a new study from scratch.  Second, existing research might be available that provides important context.  For example, let’s say you are in the alcoholic beverage business, and you want to do some ad testing for a new vodka brand. Wouldn’t it be helpful to know if a market research firm recently published a study about your competitors’ advertising effectiveness?
3. Assess your team’s commitment. In most cases, research involves a team of at least 3 or 4 people. One person acts as the project manager (which may mean managing an outside market research agency or corralling in-house resources), usually 1 or 2 as content area experts (who may contribute to questionnaire design, etc) and usually at least 1 or 2 intended internal clients are involved (the people who will use the data at project conclusion). Now considering who is on your team, do your realistically think they will give this project their time and attention? Will they contribute to decision making? Attend milestone meetings? Be supportive when the final results come in? I have seen some potentially great market research projects fail because the person driving the project received inadequate support from internal colleagues.
4. Confirm your budget parameters.  How much budget do you have? If the project hits a roadblock, is there extra budget available? Do you have enough money to realistically hire a market research agency or do it in-house?  If you do, great. If not, you may need to just dig deeper into desk research and other  data sources.

So that’s your checklist. If you can answer these four questions, you should be in good shape to make your go/no go decision:

1. Will the market research results be used?
2. Is it true that no existing data is available that can be used instead?
3. Do you have a team that will allocate enough time to the project?
4. Do you have adequate budget?

I welcome any feedback or questions. Add them here, or call the blog requests line at 508.691.6004. Thanks!