But working with any type of consulting agency requires some level of project management on your part.  There are many risks when conducting market research; know them to mitigate them.

There are several points in a market research project where you, as the client-side manager, need to be  particularly cautious. Here are two of the big ones:

1. Risk: Ignoring sample quality. If your project is one where the agency is providing sample sources, you need to know what sources and something about their quality.  Ask how many sources the agency intends to use, what they are, and their experience with each source’s quality. A professional agency will be able to give you a clear, credible response. If you get a vague, confusing response, beware.
2. Risk: Not previewing final deliverables.  Ah the big day! The research agency is sending a sharp, knowledgeable professional to present your findings. The mistake? Not looking at their presentation at least 3 or 4 business days ahead of time.  It is totally appropriate for you to have a chance to preview and approve the presentation.  Don’t do it and you risk these very real scenarios:

• The presentation includes some conclusions that your colleagues, as experts in their market, know can’t possibly be correct. There goes the entire project’s credibility, all because of 1 or 2 poor slides.
• The presentation uses language that you just know will antagonize people. I am not saying you hide bad results; I am saying that you know your company culture and know how to present “bad news” in a productive way.
• The slides contain obvious errors.  Alas, it happens. The agency was swamped with deadlines the week your report was due, and insufficient time was spent checking your slides.  I know a research director from a Fortune 500 firm who bumped into a presenter in the parking lot 30 minutes before an executive-presentation; the speaker was still creating slides in the rental car!

Want more tips? Please check out the book on Amazon, “How to Hire & Manage Market Research Agencies.” OR, access a free chapter here: FREE.

1. Do you belong to any professional associations? The answer should start with, “Yes, we are members of…”, and should finish with some combination of:

• The MRA (Marketing Research Association)
• CASRO
• ESOMAR
• AAPOR
• QRCA

If they haven’t even heard of these groups, run. Run far.

2. What risks are we likely to run into with our project? All market research studies have some level of risk. An honest, clear-thinking researcher will easily be able to tell you what risks he or she foresees, and how they plan to mitigate them.  If you hear a response like, “There are no risks,” be skeptical.  At minimum, most studies have risks in terms of sample quality, data collection timeframe, and the possibility of “bad news” from the research results. All are manageable, but they exist. And an experienced market researcher will be able to tell you how they can be mitigated.

There are hundreds of market research agencies out there. Choose wisely.

[Want more tips on hiring market research agencies? Check out the book, on Amazon:  BOOK ]

• Quality sample.  Having quality sample is critical. Avoid sources that can’t talk credibly about their recruitment tactics or list sources. You don’t want to bake a cake with spoiled eggs.
• Sufficient sample quantity. If you are purchasing outside sample, be sure you have access to enough—or at least have a fallback strategy in place in case your incidence rates are tragically low.  I have seen many otherwise brilliantly managed projects experience dramatic schedule slips because of poor sample quantity planning.
• Expert research design. Don’t sabotage a project by fielding a poorly constructed questionnaire or applying an inappropriate methodology for the given objective.  Even if you intend to do a project in-house, getting help in this area will pay for itself by avoiding wasted time and money.
• Competent data analysis. If you spend the time and money to collect beautiful data, don’t skimp on the analysis. This can be harder than it sounds, especially for projects that have had data collection problems. In these cases, you can become so exhausted with the project that by the time data analysis is due, you are just ready to be done with it. Resist the temptation or get help.  Skipping on data analysis can backfire in 2 ways. First, someone else will look at the data and find an important result that you missed. Ouch. Second, you won’t get a good return on the research investment—leaving a bad impression that may derail you the next time you try to get budget for a market research study.
• Decent reporting.  Rushing to get the research delivered is understandable—but unfortunate typos, mislabeled charts or poor writing can turn off the audience and damage the research’s credibility.  Yes, really.

If you have people with these skills in-house—great, do it in-house. If you want to augment with outside expertise for discrete tasks, there are many boutique agencies and market research freelancers available for such work. And if you want an experienced expert to run the project for you, by all means hire a market research agency. But don’t skimp—it always backfires.

[What do you think? Do you agree? Have a different perspective? Please add your comment here or call the blog comments line at 508.691.6004 ext 702.]

Do these websites matter? Aren’t they just electronic billboards?

I think they do matter.  A lot of people who buy and use market research visit agency websites. And frankly, they are more likely to visit an agency site than that of a professional association (such as the MRA or CASRO) or of a trade publication (such as Quirk’s or Research Magazine). So these agency websites are, intentionally or not, a key source of client education.

And at the risk of sounding preachy, I think all market research professionals have a responsibility to promote market research ethics, professional standards, and innovation. And one of the ways to do it is through our public faces.

The Method to My Madness

I selected 6 websites from the Honomichl list and applied my 4 criteria. To refresh, the criteria are:

1. Educational content.
2. Standards and ethics.
3. Friendliness.
4. Innovate thinking.

Grades of A-D were assigned based on specific criteria within each metric. As an example, an agency that has at least 20 educational white papers, videos or similar forms of content got an A on the Education metric. As another example, an agency that has no mentions of market research ethics or adherence to ethical standards got an F on that metric. And if I had to search for an item for an hour only to find it buried in press releases, that gets a D. Please note that in some cases I may have missed relevant content—but if I spent an hour seeking out content for a given metric and could not find any, it is for all purposes, a low grade.

So how did the agencies do? I selected the following agencies from the Honomichl list:

Burke

IPSOS

Maritz

Market Probe

Synovate

TNS

And to be clear, I am not looking to pick on anyone. I did this as an exercise to see how well agencies do on these metrics.  My key take-always:

• Educational. Incredibly inconsistent, though a few shining stars.
• Standards and ethics. As a group, agencies do this poorly. None had mentions of ethics or standards on their home page or other prominent pages on their web sites.  Professional association badges are hard to find, if evident at all.
• Friendliness. Again, very inconsistent.
• Promotes innovative thinking. Very inconsistent. A couple of agencies do a good job; a couple seem to neglect this angle altogether.

So How Did They Do?

The top 2 grades go to….drum roll….IPSOS and TNS.  Both have web sites that have the highest scores on the metrics applied. IPSOS has a bit more content that promoted market research innovation. But both sites do a great job on education and friendliness.

I welcome any feedback here, or by phone: 508.691.6004 ext 705. Thanks!

But if you have ever looked at more than 2 or 3 agency web sites, you will see that the quality and content vary dramatically. Almost dismayingly so.

So how to compare them? How to decide if an agency is worth further consideration?

I recommend using the following 4 criteria to assess market research agency websites:

Educational content. Number of white papers, videos, blogs and other content that educates client-side visitors. An agency that invests in client education is an agency that values its client relationships

Standards and ethics. Visible statements about market research ethics, and adherence to standards promoted by the industry’s professional associations such as CASRO, the MRA and ESOMAR. Choose agencies that have solid credentials; alas, there are many that do not.

Friendliness. Based on a site’s overall readability and ease of navigation. Why hire an agency that can’t communicate about research in an accessible way?

Innovative thinking. Has staff-authored books or papers on innovative topics, promotes new methods on website or blog. Perhaps even shares agency-funded “research on research.” An agency that spends some time and money investing in research methods will be able to best advise you on what methods will best meet your needs; you don’t want to go to a doctor that hasn’t been trained in 20 years, do you?

Obviously, a market research agency’s website is only one way to become exposed to its capabilities. Still, it is an important one. To find links to market research agencies and their websites, try the AMA’s Green Book, Quirk’s directory, the MRB Directory, or for a more global listing, ESOMAR’s online directory.

[This is part 1 of a 3-part blog series on market research agency web-sites: the next post will cover some of my favorite agency websites which meet the criteria above. Be sure to get the RSS feed so you don't miss parts 2 and 3: subscribe.]

Using price as the selection basis is simple and comfortable.

So here’s the catch: if you are sending RFPs only to agencies that come back to you with similar ideas, it’s time to rethink your short-list.

These days, there are innovators out there. Agencies using cooler tools, applying newer sample quality processes, and even offering new deliverables. Their methods may push you out of your comfort zone. Their proposals may be harder to read because they won’t be full of the same boilerplate you’ve been seeing for years.

But if you want research that will have an impact, break you out of a market research rut, and create more excitement among your internal clients at the end of the day, it’s time. Create a new agency short-list that includes innovative firms, perhaps even a few that you are skeptical about. You may be pleasantly surprised at the options they offer you.

[If you’d like help rethinking your agency short-list, I can help. Contact me at 508.691.6004 ext 705, or KKorostoff@ResearchRockstar.com]

[Want more market research ideas? Subscribe to the newsletter: FREENEWSLETTER]

• When you ask questions, do they listen and respond respectfully and credibly?
• When you ask for recommendations, are the responses thoughtful?
• Do they come to you proactively with suggestions and ideas?
• When you read their reports and memos, do you feel the information is reliable and credible?

I recently spoke with a long-time client who confessed that the only reason he hasn’t fired his market research agency is fear of change. For him, it’s the “devil you know…” syndrome. But because he doesn’t trust his agency, he is spending an inordinate amount of time checking their work and documenting communications via memos.  I’m working on a new short-list for him so that at least he can see what his agency options are. Will he take the plunge? I’m not sure. But I feel compelled to encourage him to look.

There are hundreds of market research agencies and hundreds of market research consultants. Changing agencies can be painful, but you deserve an agency that you can trust with your valuable market research investments.

Need to rethink your agency choices? Interested in finding agencies that offer more methodology options? Check out the directories at the MRA (http://www.bluebook.org/index.cfm) and Quirk’s (http://www.quirks.com/directory/index.aspx). Want help creating a new short-list of potential providers? Contact me at KKorostoff@ResearchRockstar.com. Thanks!

The biggest miscalculation people make when bringing research in-house is underestimating how much staff time it takes to analyze and report research results.  Even for a simple quantitative project—one without any multivariate analysis or modeling—you can easily spend 20-40 hours doing the data analysis, quality checks, creating visual displays, and preparing deliverables. And again, that’s a low time estimate.

Even if you have great people on staff with the right skills to do the analysis and reporting, you will be setting them up for failure by underestimating the time they need for such tasks. And we all know the brutal truth about primary market research; no matter how important and groundbreaking your insights are, it won’t matter if your audience doesn’t get them delivered in a clear, compelling way.

So what are your options?

• As you bring more research in-house, be sure to have a realistic resource plan in place. Create a best and worst case scenario for staff requirements given planned research levels.
• Establish a pool of qualified market research freelancers/consultants who can augment your staff during crunch times.
• Provide some basic training to your internal clients on what to expect from market research projects (ok, that is a bit self-serving since I do this type of training—but I still think the point is valid).
• Consider options for how you can best optimize the mix of in-house versus outsourced market research. For your organization, it may make sense to bring more research in-house—but choosing how to strike the right balance given your internal resources will have a big impact on this transition’s success.

[As always, I welcome any questions or comments. Please leave them here, email me at kkorostoff@ResearchRockstar.com, or by phone 508.691.6004 ext 705]

Some of the pain that market research is going through right now is precisely because of this. In the field of market research, we have hammers we are very comfortable with:

Full-service research offerings. Bang.

Quantitative research. Wham.

Focus Groups. Smack.

In-depth interviews. Bam.

The challenge is that the nails have morphed more quickly than our hammers. Now we have an increasing base of customers who don’t necessarily need full-service offerings. Or whose “full-service” needs are very different than what they used to be. Still, they get whacked with the same hammer.

I also see an increasing number of nails…I mean, customer needs…that can be met with excellence using newer tools, but many long-time research suppliers are still stuck on conventional methods. I just met a client this week who specifically shared such an experience, and how it has resulted in an “aha” moment of, “…we need to reevaluate all of our longstanding research partners to make sure they are giving us the best options.”

So for my market research agency readers, I wonder: do you have the right hammer?

And for my friends on the client-side, the fact that we are at an inflection point in terms of market research tools and applications is exactly why I recommend using RFPs these days (boy am I going to get hate mail for this!). But truly, if you are about to engage on some important research, send an RFP to a few different agencies with a few different profiles, and see what you get back. Agencies with different sets of hammers will give you some fresh perspectives. At minimum, you will get new ideas and maybe even some education. And perhaps you will even find a proposal that truly does hit the nail on the head.

I’ve done over 600 primary research projects in my career, and at least 30% have been multi-national.  So I have learned a few things.  Sometimes, the hard way.

If you are planning a multi-country survey project, here are a few tips.

1. Assume at least 5 business days in your project schedule for translation. And that is the bare minimum. It really does take time for proper translation and quality checking of that translation.
2. Hire a professional translator.  Even if your good friend Alberto speaks fluent Italian, trust me: translation is a unique discipline.  Of course, you can always ask Alberto to check the completed translation for you, as a sanity check.
3. Keep your questions as short and simple as possible. Because of language differences, a question that seems fine to you in English may translate to be more cumbersome in another language. Also, simple questions pose lower risk of translation heartache.
4. Plan for translation at the end of the project.  If you plan to have any open-ended responses at all, budget for it.  If you end up with 1,000 open-ended responses to an important question, you’ll want them translated.  And 1,000 responses, even just 8 to 10 words each, adds up fast.
5. Beware of subjective scales.  Because of different cultures in different countries, even regions within countries, subjective scales can be hard to interpret.

About Those Scales…

This tip about scales is really important.  Let me give you some examples.

“Please rate your satisfaction with our product from 1 to 5, where is Not at All Satisfied and 5 is Very Satisfied.” That’s subjective.  What I mean by “very satisfied” may not be what you mean. And in some cultures, those 5s are almost never given out. In others, they are handed out like candy. So if you are collecting data in 10 countries, and using a very subjective scale, how can you reliably compare results county-to-country?

If you are working with a full-service market research agency that has experience with the population you are researching and the countries you are including, they will be able to give you guidance on how to do those comparisons. But frankly, it’s not perfect.  So I recommend playing it safe; use subjective scales sparingly in multilingual surveys.

In the case of satisfaction research in particular, this is another reason why it is important to collect objective behavioral data as well. Data such as number of repeat purchases in past 6 months (or planned for next 6 months), number of times has recommended your product to a friend/colleague, willingness to be a customer reference, etc.

For some topics, a useful but oft-neglected scale option is constant sum. A constant sum scale is one where respondents are asked to allocate 100 points among a list of (typically) 7 to 10 items—such as desirable product features, needs, values, criteria. This gives a more objective result than listing a set of items and asking each one be rated on a 5 point scale from, for example, “Not at all important to me” to “Very important to me.” That approach typically results in everything being important—not very useful.

Bottom-line

Multilingual surveys take more time to plan, more time to execute, and require very careful question wording and scale selection. If you don’t have direct experience with them, I strongly recommend working with a full-service market research agency, or a market research consultant, with proven experience in the countries your research will cover.

Want more? Check out the webinar, stored here with audio:  LINK.

[Next Drawing for a Research Rockstar mug is tomorrow. Just add a blog comment to enter! One winner selected at random every 2 weeks from valid blog comments.]