Remarkable Research Sets Realistic Expectations with Clients

One aspect of setting client expectations is by distinguishing between confirming existing insights or discovering new insights.

• When I walk into a store because its ad promises, “The latest in kitchen appliances” and I am greeted by a display of side-by-side refrigerators and four-burner electric stoves, my initial excitement is sucked out of me. Even if I don’t leave right away, I am now on high bait-and-switch alert. If we promise research clients (internal or external) “exciting” research, but what we deliver just confirms stuff they already know, it’s a market research bait-and-switch. The client is let down. So we need to be careful and position the new research precisely. Is it stuff they already know and are we seeking to confirm or quantify it? Or is it about new strategy-altering topics? Both are legitimate types of research. But we need to set expectations correctly, or we will be viewed as shucksters.

[This is the ninth article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Research is Not About Abandoning Proven Methods

Remarkable research is not about replacing well-tested, proven methodologies (such as surveys and focus groups). It’s often about augmenting them. Alas, I see many cases where people eagerly embrace the “hot-new-thing.” I understand the temptation: it can help with research audiences (internal or external clients) who may find newer methods more interesting than the results of another “boring” survey. In reality, cool new methods are often best at augmenting the tried and true, but not always for replacing them.

• Here’s a great example: I recently worked with a client who used an idea management platform to collect preliminary feedback on some product enhancements. This led to an amazingly well planned survey as a Phase 2. Bonus: because we were able to weed out several ideas in Phase 1, the survey was nice and short—which made for a compelling survey invitation to his in-house panel (invitations to a 5-minute survey generally out-perform those to a 15-minute one).

I stand by the old aphorism: If it sounds too good to be true, it probably is. So before you abandon proven methods, do some serious pilot projects first.

[This is the eighth article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

[New dates have been added for our 4-week class, "Online Research Methods for 2012", taught in Research Rockstar's virtual classroom.]

Remarkable Research is Delivered Using a Multi-Mode Strategy

So you have a great new market research study completed. Now what? How do you deliver the findings with both impact and credibility? Take a little time, and craft a comprehensive deliverables strategy. The key components of this strategy will include a mix of delivery methods, with emphasis on retention building among the audience.

• Find ways to repeat key pieces of information in different ways and times. I have had the experience where I repeated information, and a key audience member only “got it” after the third or fourth exposure. Were they not paying attention earlier? They probably were. Some people “click” with charts, others with stories or anecdotes, others with video. If they get the information the day they are on a deadline or if they are dealing with another matter, they may intend to pay attention but are not able. Repetition using multiple approaches at multiple occasions is critical. As a researcher, you may feel repeating and repackaging information is redundant, but it isn’t. What seems obvious to you is not always obvious to an audience.
  • Tip: This point about time is important. Yes, the project is done and you want to deliver it right away. But by repeating key parts over a few weeks, you will maximize the chances your audience will use the research.

Research involves delivering results. Remarkable research involves making sure our audience actually uses them. If we expect people to retain and apply the results, we have to create a more comprehensive delivery strategy. Presentations, email follow-ups, executive endorsements, podcasts, custom follow-ups, 1:1 briefings, posters, handouts and internal blogs are some of the options that can be combined to make sure our audience is reached multiple times using multiple modes. Lather, rinse, repeat.

[This is the seventh article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

[New dates have been added to our online, real-time classes taught in our expanded virtual classroom: Check it out!]

Remarkable Research Includes Listening, Not Just Asking

Remarkable research is closer to our fingertips than ever before.  And it often involves listening, not asking.

One of the most efficient options is through social media research. By monitoring social media conversations using tools such as those from BuzzMetrics, Crimson Hexagon, NetBase, Radian6 and Trackur, researchers can discover customer attitudes without having to ask. This dovetails with the current hot topic of behavioral economics, which among other things, points out that research participants cannot always report their attitudes and behaviors accurately. For some topics, listening and observation is the key to gathering accurate data.

Ethnography is another option for projects where observing will yield better insights than asking. But be careful: ethnography is a nuanced discipline.  Videotaping a bunch if interviews is not ethnography.

[If you’re interested in learning more about social media market research, click here to read about our online class.]

[This is the sixth article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Research Doesn’t Have to Include a Remarkable Price Tag

Market research exists on a continuum ranging from “quick and dirty” to “meticulous and complete.” And these days, there are some quick and inexpensive methods that aren’t that “dirty” at all. Before assuming that a new market research need requires a 3-phase, qual-quant methodology supported by a team of expert moderators and stats PhDs, ask yourself: what are all of the options? If you do decide a research need can be met by a quick, inexpensive approach, consider:

• Secondary research. More resources are available than ever. Don’t reinvent the wheel—see what exists; you may be surprised.
• Social media research. Using one of the free or lower cost tools (as opposed to some of the more elaborate tools which do come with bigger price tags), you can get directional information on what the buzz is on brands, products, cultural trends, etc.
• Facebook or LinkedIn polls. They are free and can be fast (assuming you can push that poll to a large enough network).
• New tools. Many new services integrate online research tools with panels. Check out AYTM (disclosure: Research Rockstar does have a relationship with AYTM), Zoomerang, and SSI (via its QuickTake offering).

Sure, some projects warrant the big bucks. But when big budgets and long timelines aren’t an option, useful information can still be gathered using creative methods.

[This is the fifth article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Research Requires Qualified Participants

For many market research projects, success can be foiled by unqualified or disengaged participants. I have seen many online survey projects start with an excellent intent, stellar questionnaire design…and then wasted by weak participants.

• One option: companies can build an in-house panel to prepare for remarkable research. For certain target markets and populations of interest, this can reduce data collection costs and pay for itself quickly. Can initial investments to build a panel be high? Yes. But of course, it is all relative. How much time and money do you waste by collecting data from weak sample sources? How many times have results been questionable—because the participants’ qualifications or authenticity was dubious? Panels cost less to build and maintain than MROCs, and in many cases deliver the needed ingredient for successful research.

[This is the fourth article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Research Isn’t Just About Asking the Right Questions; It’s About Making It Easy for Respondents to Give You the Right Answers

Remarkable Research isn’t just about asking the right questions; it’s about making it easy for respondents to give you the right answers.

Anyone who designs online surveys quickly learns that to get good data, you need to take a hard look at how you choose your answer options and scales. Avoid over relying on 5-point scaled questions and consider all of the options: semantic differential, constant sum, rank order, even open-end. There are many options, but don’t go too crazy. A patchwork of many different answer options and scales in a single survey becomes burdensome for participants.

• Tip: Another great resource is The Handbook of Marketing Scales.

[Want to learn more about choosing answer options? Check out the "Ask It Right" class from Research Rockstar. Also available in our virtual classroom on January 23rd and February 16]

[This is the third article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Research Requires a Commitment to Continuous Improvement

Remarkable research requires assessing every completed market research activity. What worked? What didn’t? Were the clients (whether internal or external) satisfied? Were outside suppliers satisfactory? One way to do this is through a “post mortem” evaluation of the research process.

• After each project, conduct an objective, step-by-step review to find opportunities for improvement. After that, create an at-a-glance reference for next time.
  • Tip 1: Quickly capture post mortem information, preferably within a week of project completion before memory fade sets in.
  • Tip 2: Craft a standard project assessment form to make it easy for you and team members to record key information. Just don’t make it onerous (else it won’t get used). 4-6 questions is sufficient.
  • Tip 3: Review all assessment forms at least once a year to look for recurring challenges (or if you prefer, “opportunities for improvement”). Sometimes these trends are not so obvious during day-to-day project management.

[This is the second article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.]

Remarkable Researchers Offer Their Clients Options

If a client comes and asks for a survey, it’s up to the remarkable researcher to look at the project goals and recommend methodology options. While surveys and focus groups are great, remarkable research can leverage a diverse range of methods. When we offer clients options, we benefit in three ways:

• We thrill the client.
  Especially, if the client grudgingly uses surveys as a better-than-nothing option. Everyone loves options. If I go into a bakery and there are only two types of cupcakes on display, I am immediately turned off. If I walk into one with too many choices, I may feel overwhelmed and leave. Neither too few nor too many options are the goal.
  
• We educate the client.
  It is important to raise awareness of alternate methods. This is so business professionals don’t see market research as unremarkable, old fashioned, or equated solely with surveys and focus groups.
  
• We demonstrate integrity.
  We aren’t just offering what is easy or conventional; we’re showing that we are thinking about the client’s needs. Even if they choose a survey option, now they know we were looking out for their best interests.

This is the first article in a series of 20 mini-posts titled, “Principles of Remarkable Research.” Don’t want to miss this series? Subscribe to our blog via email or RSS.

“So now our staff has been trained in market research best practices, but how do I remind them? I suspect that once they get caught up in daily work, and urgent deadlines, these lessons will fade from their minds.”  This question came from a Research Rockstar client, and it raises a very valid point—training is not a one-shot-vaccine-for-life.  We all need booster shots. So to help this client’s staff keep best practices in mind, we crafted a gentle “refresher” program of peer reviews, mini-classes and simple email reminders. This prompted me to create a general series of best practice mini-posts, which I’ll be sharing here.  The new “Principles of Remarkable Research” series starts tomorrow, and is a series of 20 reminders, refreshers and simple lessons. These are super short—all are under 200 words! Please share with your friends and colleagues.

Don’t want to miss this series? Subscribe to our blog via email or RSS.

Click here to subscribe to “Market Research Training from Research Rockstar” via RSS

Click here to subscribe to “Market Research Training from Research Rockstar” via eMail

[If you haven’t done so already, click HERE to sign up for Research Rockstar’s Market Research Newsletter.]

December can be a nice, calm month for market researchers. Typically a month with little data collection, few presentations and no travel, it’s an ideal time to do a little 2012 planning. Here are three tips to keep in mind:

 
• Get customer feedback on 2011 projects. If you haven’t done it yet, it’s time to practice what we preach. Ask for some honest feedback on recent research. What worked. What might want to be repeated in 2012. What requires further investigation.
  
• Get aligned with 2012 corporate initiatives. Organization planning major international expansion in 2012? Planning to deploy green practices? Reducing time to market for new products? Whatever it is, now is the time to plan proactively for how market research can support key organizational efforts.
• Get sample sources updated. Whether you rely primarily on an in-house market research online community (MROC) or panel, have a regular sample provider or some combination, now’s the time to plan ahead for 2012 sample needs. Does your panel need refreshing? Does your MROC need expansion to cover new market segments? Have you had sample quality concerns in 2011 that need to be addressed before 2012 projects start? It’s a common source of market research headaches, but better to plan ahead.

Kathryn Korostoff will be presenting a one-hour webinar on Thursday, December 15, 2011 at 12:00 pm (CST). This webinar is ideal for anyone interested in learning emerging market research techniques.

Think Outside the Survey: 14 methods to change how your colleagues and clients perceive market research

Based on the new eBook, “Think Outside The Survey”, Kathryn Korostoff will discuss 14 specific methods that will change how you and your colleagues and clients think about market research. Too many people dismiss “market research” because they equate it with surveys and focus groups. Our goal is to get them to understand that market research is more, much more. Korostoff will present 14 market research methods that aren’t surveys or focus groups. These are the methods we need to raise awareness of, so that people, won’t dismiss market research – even if they are dismissing surveys.

Some of the methodologies discussed include:

• Biometrics
• Crowdsourcing
• Mobile Research
• Neuromarketing
• Video Research
• + 9 more methodologies

There will be a Q&A session at the end.

Let’s spread the word that it is time to think outside the survey. Let’s start by educating ourselves about the new methodologies. Click here to register.

This webinar will be hosted by Quirk’s editor Joe Rydholm and is sponsored by Zoomerang.

Are you tired of people being cynical about market research? Tired of explaining to people that market research is, in fact, much more than surveys and focus groups?

Me too.

Announcing, “Think Outside The Survey”, a new eBook from Research Rockstar. Members can access it on the Members’ pages. All others, please click here.

Today, some business professionals dismiss market research. Thanks, in part, to popular books like Predictably Irrational, they have learned that self-reported behaviors and attitudes can be unreliable.  But the problem isn’t that all market research methods deal with these realities poorly—the problem is that many people assume market researchers rely exclusively on surveys and focus groups. And while these are great methodologies useful in many situations, they are among the most susceptible to the deficiencies of self-reported behaviors and attitudes.

Our goal, as market researchers, is simple: we don’t want business professionals to dismiss “market research” because they equate it with surveys and focus groups.  Our goal is to get them to understand that market research is more, much more.

That’s why this new eBook presents fourteen market research methods that aren’t surveys nor focus groups. These are the methods we need to raise awareness of, so that people, won’t dismiss market research—even if they are dismissing surveys.

Please share this eBook with your friends and colleagues. Let’s spread the word that it is time to think outside the survey.

Part of being a good market researcher is the ability to determine when the research conclusions we start to draw reflect reality, and when they’re only coincidences. A few shiny data points can easily tempt us to jump to conclusions too quickly—before we dig deeper.

I’m reminded of raising a teenager, who reported that she “just had to have” a certain pair of shoes. They were stylish. They were unbelievably comfortable.  And all her friends were getting them. In good parental fashion, I struck a deal. We’d wait a month until report cards, and if her grades met expectations, we’d consider the purchase of $100 shoes.

A month later the grades were fine, but the shoes were no longer desired. Turns out nobody was wearing them anymore because they were too tight, fell apart and looked funny. Yet had we gone with her initial “analysis”, we’d likely have concluded that this was a great product — and we’d have been wrong. So how do we avoid the trap?

Market Research & Insight

As researchers we have to keep ourselves honest.  A few coincidences do not a key finding make, and I’ve seen some frightening cases of “conclusions” based on weak data analysis and flimsy proof points. This isn’t value—it’s hyperbole.

During analysis we can’t assume that “interesting patterns” reflect a broader reality, nor force a conclusion where none is justified.

Here are two best practices to help practice the discipline of meaningful insight creation:

Challenge your own conclusions

If you’re doing a research study and the data seems to perfectly support all your hypotheses, you need to ask yourself whether your research or the process was biased.  Perhaps your research was intended to confirm things your organization already knew. But if you’re hoping for some real “aha” points, this may not be what you want.  If your research results wrap up too neatly, look closer.  Is there an opportunity to challenge any of these results?  I’ve done several customer loyalty studies that at face value gave very positive results—customers were broadly satisfied and loyal. Digging deeper, though, we learned that important segments were loyal, but due to existing contracts and cost-of-switching issues — externals that dramatically changed the story.

One way to challenge your conclusions is to seek contradictions.  Look at how specific key attitudes and patterns vary by subgroup.  Examine differences by gender, age and geography, or if it’s a B2B study, by job function, industry and company size. Does the story still hold?  Sometimes the obvious is masking something that might require a bit more intellectual rigor.  As researchers, it’s our job to be disciplined and get beyond face value.  What’s the story behind the story?

Validate your results

We have to validate our data, especially when the research conclusions seem too-good-to-be-true. Whenever possible, validate by comparing with other data sources. You want to determine if relevant external data points also show consistencies about the customer attitudes and behaviors you’re examining in this particular study. A couple of examples:

• In a recent survey project, the client used social media monitoring to find customer stories and anecdotes that reflected behaviors measured in the survey project. Not the most rigorous methodology, perhaps, but it did provide real-world examples.
• In many studies, I have had clients doing online surveys conduct a small number of follow-up IDIs with respondents to access some of the “why” context.

Market Trend or Coincidence?

Sometimes as researchers we get so excited about our data that we jump.  But remember, a few coincidences don’t justify a conclusion.  We have to make sure that the story we’re sharing can be confirmed, replicated, and demonstrated with solid proof points. Otherwise we must present those conclusions as hypotheses, or “directional”. That’s not necessarily a bad thing, but it is different, and we have to have the discipline to recognize the difference. Otherwise we’re just peddling bad shoes.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Fortune 500 researchers often juggle the need to deliver fresh customer insights with the mandate to minimize research costs. How do they do it? By cutting costs where they can, and embracing cool new technologies when applicable. Here are three strategies currently being embraced by Fortune 500 market researchers.

#1: Market Research Using In-House Panels

Companies often rely on third-party panels as a sample source for survey research (for example, you may use EMI, SSI ort USamp for your online surveys). Third-party panels offer appealing convenience and predictability. Still, if your research requires focusing on your own customer base or special screening criteria, third party panels may not be the most cost-effective choice. As a result, some companies have invested in building their own in-house panels. For certain target markets and populations of interest, an in-house panel can reduce data collection costs and pay for itself quickly.

Will an in-house panel be a good fit for the types of research your company does?

• Your participants will know who is sponsoring the research, and that does introduce some bias (you are more likely to get panel members who already have strong awareness of your brand and even a preference for it). Is that an acceptable trade-off to your organization?
• If some of your research needs to be with more random populations, you need to ask yourself, “Are my panel members an acceptable proxy for the broader target market?” Or, will you have budget to augment those studies with a third party panel?

Of course, if you do a lot of online surveys with your customer base, it’s more of a slam dunk. In these cases, it makes sense to really mange your customer list as a panel, by giving them the option of opting in to a panel program, and tracking their participation.

#2: Augmenting Traditional Market Research with Social Media Insights

Many market researchers now accept social media-gleaned insights as a way to inform market research projects. By monitoring social media conversations using various tools such as Buzzmetrics, Crimson Hexagon, Radian6 and Trackur, corporate researchers can discover trends in brand sentiment and even gather product feedback without going out and asking for it. While in many cases, this type of research is viewed as more “qualitative” and directional, as opposed to “quantitative,” it does have value. The large amount of social media content that gets generated worldwide every day is a rich source of data that can be analyzed using cool new technologies (in the form of text analytics and sentiment analysis tools). Opinions are divided about how best to use the data, but many corporate researchers are embracing it at minimum for “discovery” studies as a Phase 1 (to inform a more significant survey project as a Phase 2) and many use it for general WOM or buzz monitoring (often as an early warning system).

To learn more about social media research, please download this white paper from Research Rockstar.

#3: Seeing the Future: Prediction Markets as a Market Research Method

Some Fortune 500 researchers are starting to test prediction markets as a market research method. A prediction market is simply a web-based platform to generate, prioritize, and assess predictions. Want to know which of several new products will sell more? Maybe you want to know what behaviors will be more common in your target market by 2015. How about finding out brand perceptions by asking which of your top four competitors will have the most revenue growth next year? Ask the crowd, whether a broad or narrow one, by hosting a fantasy stock market or “poker chip” game. IdeaScale, Infosurv, and Inkling are just three of the platforms that offer trials. Again, new technologies are allowing corporate researchers to gain customer insights quickly.

Fortune 500 Market Researchers Spend Research Dollars Wisely

Just because they work for big companies, it doesn’t necessarily follow that Fortune 500 market researchers have big budgets. They are just as eager as any other researchers to gather as much insight as possible while managing expenses. Today many are starting to take advantage of new technologies to do that. Still, it’s not about replacing well-tested, proven methodologies (such as surveys and focus groups); it’s often about augmenting them.

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

10 things great market researcher project managers do:

1. Challenge all assumptions
2. Document deliberate project goals
3. Think precisely about sample sources and screening criteria
4. Plan for contingencies
5. Manage client expectations (internal or external clients)
6. Identify recurring themes or patterns of data and insights
7. Weave individual findings and data points into key take-aways
8. QA ALL data before distributing
9. Create deliverables that are appropriate to audience needs and abilities
10. Distinguish between confirming existing and discovering new insights

Regardless of qualitative or quantitative methods, great project managers do these things. Some do them by instinct; others do them according to formal project management frameworks. But they all do them.

Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter.

Note: this is the third and final part of a series on transitioning to in-house market research. (Click here to view Part 1 and Part 2.)

Weighing the trade-offs between in-house versus outsourced market research is more complex than it might appear at first glance. As we have already talked about in this series, there are some compelling benefits, and thanks to new technology, real feasibility. But even after you assess the tools, the trickiest part is still at hand: how to assess in-house skills.

Can In-house Staff Handle All Market Research Needs?

Every organization is unique. In some, the plan may be to train existing staff. In others, the plan is to hire new staff. And in still others, a combination of training, hiring and even planned use of market research consultants.

Regardless of which strategy you plan on, the real challenge is to realistically gauge the amount of work and skill levels you will need—should you go so far as to fire your market research partners (well, “fire” may be extreme, but “downgrade” may apply).  A pilot phase, as recommended in Part 2, is the best way to do this.

In assessing your staff’s readiness for more in-house work, there are certain skills that are basic requirements. Depending on the types of projects you conduct, knowledge of basic market research methods and techniques are obviously important.  But beyond the basics of questionnaire design, interviewing techniques, recruit management, statistics and so on, are several categories of management skills that your market research agency partners have been providing. The question is, can your staff take on these aspects as well?

Market Research Skills Assessment

As you conduct 1 or 2 pilot projects in-house, assess your staff with brutal honesty on the following five items.

• Project Management Skills: Running a market research project, of any kind, from start to finish includes coordinating a lot of moving parts. Even in its most basic form, project management can be a very intense job, especially for the many real-world market research projects that have tight budgets and firm schedules.  Q: During the pilot project, did your in-house team do a great job on keeping to a schedule, managing a budget, and communicating with internal clients?
• Reporting Process: Of course, it’s not just about project management. You need to go through an entire project and deliver results upon completion. What kind of output is generated? What does it take to turn that output into a client-ready report?  Does your in-house team realistically have the time to focus on generating great reports? Even if it’s for a “friendly” internal client, was the in-house team equipped and able to generate a final report?  Q: Does your in-house staff have enough experience with market research tools and analysis techniques to create a report that gets to the real “so what” of the results?
• QA Skills: Market research companies often have rigorous best practices for quality control (how they clean data sets for quantitative projects, how they error check charts, and so on).  When you do your pilot projects, see how well your team manages the quality aspects. Are results delivered free of errors? If there is data that looks contradictory, does your staff identify those issues proactively, before your internal clients do?  Q: Does your staff have the skills and processes to check and enforce data quality?
• Presentation Abilities: It’s one thing to write a market research report; quite another to actually present the results.  The last step in many market research projects involves making a presentation to executives, and that requires the ability to stand in front of a group of people and present research in a compelling, objective, credible way. That ability can be boosted by context gleaned from having done many projects over many years. For example, a presenter who can truthfully say, “This unexpected finding is, in fact, consistent with other messaging studies I’ve done where…” is using experience to add context, and this enhances the research’s credibility. When you do your pilot projects, be honest:  do in-house staff have the required level of credibility? Do they engage their internal colleagues sufficiently to get them to really pay attention and use the research results?  Q: Are your in-house staff perceived as credible when delivering research results?
• Delivery Effectiveness: The brutal reality is that sometimes an outside research firm can deliver the bad news that market research studies may uncover better than in-house staff.  You may find that customer satisfaction isn’t what you’d hoped, or that a new product idea isn’t being well received.  Often, it can be easier for an outsider to deliver that news—and easier for the audience to “hear” it from them. So even if you have in-house colleagues who are amazing presenters, it doesn’t necessarily mean that they can be effective with the tough projects. Q: Will your internal staff be comfortable reporting bad news to their senior-level colleagues? And will those colleagues really listen?

Bottom line: Assessing your staff for basic market research skills is pretty straightforward; questionnaire design, interviewing techniques, and statistics skills are all obviously either there or not. Assessing your staff on the skills beyond the mechanics can be trickier—unless you do an actual pilot.

If you assess your staff skills across these areas during a couple of pilot projects, you’ll have a good sense of whether or not you’re indeed ready to move more towards in‑house market research. And, more importantly, you’ll know what staff skills to invest in or augment.

(This concludes a 3-part series on transitioning to in-house market research.)

Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter. See our current online class list too, many are under an hour.

As you begin to reconsider your mix of outsourced vs. in‑sourced market research, it is easy to get swayed by the very real, potential benefits: feasibility, responsiveness, security, costs.

But even with compelling reasons for moving toward in-house research, proceed with caution.  In fact, the best option is to take a pilot approach. Take two or three months, and a couple of in-house projects, while you test the tools you plan to use.

Test Your Market Research Tools

If you’re going to do more research in‑house, then you’re likely planning on investing in some specialized tools, whether it’s an online survey platform, a panel management program or social media research tools.  Many of these products come with trial periods, so take full advantage of that opportunity to actually use them.

As you try-before-you-buy, ask yourself some honest questions:

• How easy are these tools to use?
• What investment, in time and money, will you have to make to get your staff proficient in their use?
• Do these tools really have all the features that you need?
• How much time does it take to set-up or program the tools for real-world research?
• How much time does it take to customize or automate reports or other output?
• Will your in-house staff have the time or skills to manage these tools, or in reality, will you have to engage with a consultant who specializes in them?

Until you use these tools for a “live” project, you can’t really assess either the learning curve investment or the true “cost” of using them.

Market Research Tools Take Expertise

A market research agency typically has staff with extensive experience with various tools. Indeed, it is common for a market research company to have staff who specialize in programming with specific online survey tools, and so on. If your in-house volume of work is low, it can be hard to develop that expertise in–house. It’s hard to maintain—let alone build—expertise with specific market research software products when you have weeks or months between programming tasks.

Bottom line: Yes, there are many amazingly simple tools available to facilitate the research process these days, but they still take time and money to acquire and use. So before you transition to in-house research, be sure you honestly assess the true costs of these tools—both in terms of actual out of pocket expense and staff time. By testing tools out during a trial period with real projects, you will reduce the risk of having overly optimistic expectations.

Coming Next: In the third and final part of this series on transitioning to in-house market research, we’ll provide a checklist for realistically assessing in-house skills and staffing requirements.

[In case you  missed it, Part 1 of this series can be found here.]

[Want easy, convenient market research training for your staff? See our current online class list.]

These days, it’s not uncommon for organizations that once relied heavily on outsourced market research to re‑think that practice. There are four primary reasons why you might be driven to consider bringing some, even most, of your market research processes in-house:

1. Feasibility. There are over 50 online survey platforms alone, plus a plethora of social media listening tools, text analytics programs, and more. By streamlining many once time-intensive tasks, these tools make it feasible to do research in‑house.
2. Responsiveness. There’s an ever-increasing pressure on the market research department to deliver customer and market insights faster. Often, this can be done more quickly in‑house, rather than going through a purchase process for each new project.
3. Security. In organizations that are driven by innovation — especially those related to technology and other proprietary intellectual property — there can be a concern about information security.  While not implying any unethical behavior, some companies prefer to avoid working with market research companies that also work for their competitors.
4. Cost. In some cases, it can be less expensive to do research in‑house. Out-sourcing to a market research company adds sales commissions and project management fees, and data collection costs often include a mark-up. By handling market research in‑house, you can avoid some of that extra cost.

Bottom line: Yes, there can be real benefits to re-thinking your mix of outsourced to in-house market research. Recent technology developments have fundamentally changed the cost-benefit balance. Still, there are real costs to bringing market research in-house, and some of them are easy to under-estimate. In the next article in this series, we’ll tackle the issue of assessing the real cost of market research tools. And in the third and final part, we’ll provide a checklist for realistically assessing in-house skills and staffing requirements.

[Want easy, convenient market research training for your staff? See our current online class list]

With the summer season upon us, along comes a most welcomed relief for many market research project managers. The workload slows and creates a great opportunity to take care of some market research housekeeping. For survey writers, one of the best uses of slow time is to spend it creating (or updating) standard survey templates for use year-round.

Without the normal crush of deadlines, market researchers can create templates with these goals in mind:

1. Set standards.  Use this time to think carefully about how you want to standardize specific question types, along with formatting and scaling options.
2. Get approvals. Get approval from those colleagues or managers who will be involved in year-round research efforts. Explain you are constructing standard templates and want their input during the slow season—this will help everyone come crunch time.

For those newer to research, you will find that there are three common survey templates that come in handy.  A few examples of question types are included below, but there are many options—so be sure to look at several examples before you craft your templates.

Survey Template #1: Customer Satisfaction Research

A simple satisfaction survey would consist of four or five questions to gauge satisfaction and loyalty. Of course, the type of customer satisfaction and loyalty questions depends on whether you’re selling business-to-business or business-to-consumer, whether you’re selling services or products and so forth. In general, you’re going to want an overall gauge of satisfaction, which commonly uses a five-or seven-point Likert scale question. Add a few follow-up questions about satisfaction relating to the specific aspects of your products or services, as relevant, such as customer service, product’s ease of use, and possibly aesthetics.

As an example of how this might need to be modified for different product categories, let’s consider a snack product company. For this case, satisfaction measures might focus on the variety of flavors offered, response to specific flavors, and package size.

Survey Template #2: Website Feedback

If your organization interacts with customers on its website, it’s good to have a standard template for collecting website feedback. This could be used on either a transaction or a rotation basis (so that customers see it on every 10 or 20 visits), or maybe it’s something you will use once a quarter.

Common questions collect feedback on overall attractiveness, distinctiveness, and ease of use. So answer options might use a scale of “very mundane” to “very exciting”; or maybe a range from “amateurish” to “very professional”; or perhaps, “not at all easy to use” to “very easy to use”.

Other key questions may include:

• “Were you able to find the information you were looking for on our website?”
• “How likely are you to visit this web site again in the next 30 days?” (or whatever timeframe would make sense for your particular category).

Survey Template #3: Customer Service Transaction

You may want to have a survey that’s triggered every time someone completes a support call or other type of customer service transaction with your organization. This could be done through a call center, email, or even through a social media interaction such as on a Facebook fan page or via Twitter.

Typical questions ask about:

• How quickly they got a response, which gives you an objective assessment of whether or not it was timely
• Their satisfaction with the timeliness of the response
• Their satisfaction with the quality of the response

Moreover, also use it as an opportunity to make sure that the matter was completed successfully so that you can create a red flag if necessary

Market Research Planning

In my town, the Department of Public Works knows that we’re going to get a lot of snow each winter, and you can be sure that for the couple of months before winter comes, they’re stockpiling salt and sand for the roads. They’re using that relatively slow period before crunch time hits to make sure they’re prepared.

Well, it’s the same thing for survey writing. Let’s take advantage of this slow time to make sure that we have everything we’re going to need before the next crunch time hits — as it inevitably does.

Click here to check out Research Rockstar’s full line of Online Market Research Training Classes.

Join your Research Rockstar friends for an exciting day of free market research training at MRXU, the free market research Twitterversity on July 28, 2011 (7am-3pm EST).

This is a Twitter-only event where mini-lessons are released with the hashtag #MRXU. It’s a great way to capture market research facts, definitions, and best practices. This day’s topic will be, “Principles of Market Research Project Management” and will cover practical how-tos (and how-NOT-to’s) for anyone managing a market research process. There will also be a Q&A session at the end of the event.

Come one, come all. You don’t want to miss this free market research learning event!

This next MRXU will be particularly useful for newer researchers, so please do share this info with anyone who could benefit from market research training.

Agenda Details:

Class: Principles of Market Research Project Management
When: July 28, 7 am to 3 pm EST.
All times below are EST.

7 AM: Professor Kathryn Korostoff opens the event with, “A Question of Time: Setting Realistic Time Expectations with partners, colleagues and clients.” How long do different aspects of the research process really take?

8 AM: Professor Diane Hagglund will focus on, “Special Considerations for B2B Projects”, and will be available for Q&A via #MRXU

10 AM: Ten Project Management Tools.

11 AM: Professor Greg Timpany on, “Project Management 101: Tips for Beginners”, and will be available for Q&A via #MRXU

12 PM: Professor Michaela Mora on, “A Step by Step Guide to the Market Research Process”, and will be available for Q&A via #MRXU

2 PM: Hot Topics & Final Q&A. The Twitterversity will close with any follow-up on hot topics from the day. All POVs welcome!

More details coming soon!

If you are not already on the MRXU mailing list, sign up here for updates and post-event summary access. You can unsubscribe at any time.

We all know what the most common survey design mistakes are: having surveys that are too long, too onerous, or that have questions that are leading or biased in some way.  But what about the next most common survey design errors?

1.   Too many open-ended questions.  Very often, when we’re writing a questionnaire, we realize that there are opportunities to discover new things or we are unsure of which answer options to offer. Our solution? Open-ended questions. We might ask people about unaided brand awareness; that is, “When you think of Product Category X, what brands come to mind?”  Or you might ask, “What else can our company do to improve your satisfaction with our services or products?”  Or you might have a question followed by a list of possible answers, including an “Other. Please specify:                      ” option. The first two examples are open-ended questions, but even the third one is expecting a lot.  Having two or three questions that require real writing is fine.  But if you ask too many, it becomes a turn-off. It is simply too onerous and few respondents will type that much.  The result? You end up with a lot of missing data.  So choose wisely, and use open-ended questions judiciously. 

2.   Excessive jargon.  If you’re doing a survey project, chances are that you have a great deal of expertise in a particular product category, industry, or topic area. And by virtue of the fact that you’re an expert, you have developed a specific language for talking about relevant issues. It’s very easy for those of us who develop areas of expertise to forget that other people simply don’t use the same language to discuss the same topics.  We have to be vigilant when we’re creating surveys to use friendly language.  Go for the lowest common denominator in terms of who’s going to be taking your survey—and use language that they are likely to use.  Excessive jargon turns people off and leads to dropouts, or worse. If they don’t really know what a term means they might guess, and you might be getting inaccurate data in return. 

3.   Forgetting your manners. It sounds trite but it is really true.  We need to be respectful of the people who are taking our surveys.  An occasional “please” and “thank you” goes a long way.  In the survey opening, use polite text to set the context and invite them to the survey.  Remember, they’re doing us a favor.  At the end of the survey, there should be a clear and distinct thank you message, especially if this is a survey going to your own customers.  I’m stunned at how abrupt many surveys end.  If I’m a customer and I’ve just given you 5, 10, or maybe even 15 minutes of my time to answer your survey, and it simply ends at the last screen, that’s not really very nice. 

Here’s some possible text:  “Thank you. Your opinions are very important and will help us to improve our products and services.”  Or, “Thank you. Your input has been extremely valuable.  Stay tuned to our company newsletter to hear how we’ll be applying these important research results.”  Let them know that it wasn’t just an academic exercise; that you plan to actually use the research. 

While it is great to see that there are so many free and low‑cost survey tools available today, such as Ask Your Target Market, SurveyGizmo, QuestionPro, SurveyMonkey, and Zoomerang, there are lots of mistakes that people can easily make when writing surveys.  Finding a great tool may be easy these days, but writing a great survey is not. 

Planning to write a questionnaire? Let Research Rockstar show you how to manage the process and avoid common mistakes. Click here for details.

Let’s get a few facts on the table, shall we?

1. We all have non-researcher friends, colleagues and acquaintances who are writing their own online surveys using one of the many free or low-cost tools.
2. Despite all of the well-publicized voices shouting, “The survey is dead!”, “DIY” survey volume continues to surge.
3. We may tire of hearing, “The survey is dead!”, but we all know that the role of the survey isn’t what it used to be, and we really aren’t that sad about it. Indeed, many of us now spend more time on non-survey methods, either to develop best practices or for “live” projects.
4. Short surveys are better surveys.
5. New tools and technologies create possibilities for survey design and data analysis that never existed before.

So, what does this hodgepodge of facts mean?

The role of the survey is changing. Who creates them, how they are designed, when they are used, and their role relative to other methods is all shifting, fast. To remain effective in this new reality, survey research best practices must change, and the time is now.

New Rules of Survey Design

Some of the New Rules are driven by length, others by what is feasible given recent advances.

Then: Three to five screening questions were acceptable.

Now : One or two. Better yet, none—just collect the profiling data from your MROC database or panel provider.

Then: 15 to 20 minutes average duration was the old “short” compared to the 40 to 60 minute monsters of the old days.

Now: Ten minutes or less.

Then: Multiple questions per screen to minimize number of total screens.

Now: Shorter surveys make it worthwhile to exploit the visual appeal of one question per screen.

Then: Two to three open-ended questions at most, due to both respondent compliance and analysis headaches.

Now: Wildcard. We still want to avoid demanding too much “work” from our respondents, but these questions have become much easier to analyze. Whenever an open-ended question could replace a question with a long list of answer options, it’s worth a look.

Then: Survey design and programming were done by a professional researcher, going through multiple iterations, often taking days if not weeks.

Now: Basic survey design can be easily done by any motivated professional with the help of training, survey templates, or other support.

Then: Surveys designed with the assumption that they are the primary data source for an overall project.

Now: Surveys designed with the context that the survey is only one source of data, and is likely to be used in combination with multiple data sources.

Then: Surveys designed with the assumption that there would be one mode of data collection (typically phone, online or paper).

Now: Surveys designed for multiple modes of data collection, possibly including mobile.

I am generalizing here, and there are plenty of “buts” and “exceptions.” The reality, however, is that the landscape of survey design has changed. We can’t let yesterday’s best practices hold us back today.

Bottom Line

Those of us who remember spending hours (even days) to craft what can only politely be described as visually bland surveys can be forgiven a secret twinge of envy when we see DIYers creating visually stunning surveys in under an hour. But we have to get over ourselves; new tools are in, and the old rules are out.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Once you have taken the time to craft a fantastic questionnaire, the challenge becomes: how do you get people to actually take it? Of course, having a great list (or access to an alternate sample source) is key. But so is the way in which you present your invitation. Think about the last time someone sent you a survey invitation. What questions came to your mind before you decided yay or nay? I’ll bet it was the following three questions—even if you did not articulate them this way:

1. Why should I take the time to complete a survey? You should tell them, and be earnest. Let your participants know how much you value their input. Let them know how the data will be used. Key phrases include:

• “Your input will help us to improve our services.”
• “Your feedback will help us develop new products for clients like you.”
• “At company X, we are always seeking new ways to improve customer service. And the best way to do that is to ask the experts—our customers.”

2. What’s in it for me? Offer an incentive. Yes, it helps. The promise of future products that are aligned with their needs is nice, but a bit intangible. If this is a survey of your own customers, offer them a coupon or discount code. Other popular items include Amazon gift cards and drawings for tablet computers. Note that different states and countries have different laws about contests and drawings; be sure to get appropriate legal advice.

3. Is it going to be annoying? Let them know up front that you have a nice, short, respectful survey (and be sure that your survey is just that).

Of course, the invitation text is only part of the puzzle. For more tips on maximizing response rates, see Jeffrey Henning’s excellent article from last March, which cites the work of Weimiao Fan and Zheng Yan, and another one from May citing the work of Joel David Bloom, Ph.D., from the University at Albany .

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Technology is fundamentally changing how, when, where and with whom we do research. These changes fundamentally challenge the market research industry’s tired dogma of research being “great, cheap and fast—pick any two.” And this, in turn, challenges current market research agencies, suppliers and client-side market research departments.

Why can’t research be great, cheap, and fast? Low-cost tools now have high-end functions. New sample sources reduce the cost per complete and in some cases give access to more qualified participants. I’m not saying all research can or should be great, cheap and fast. Sometimes it’s fine for a research project to be “quick and dirty.” In some cases, fast is simply not an option given the quality of analysis and deliverable needed (though “fast” is relative and is faster than it used to be). However, clinging to old dogma is, well, getting old.

My biggest concern with all of this is that market research may miss the boat.  If we don’t get over our dated assumptions and embrace radical change, here is what will happen:

• Non-market research departments will have ownership of new data sources and develop their own analytic approaches.
• New breeds of consulting firms that offer new fee models, technology, and delivery mechanisms will squeeze market research agencies out.

In other words, if we don’t challenge old assumptions about great, cheap and fast, someone else will.

Note: this is an excepts from our eBook. Get it here: http://www.researchrockstar.com/market-research-predictions-paper/

The final presentation, when we deliver market research project results, feels like a huge accomplishment. The project is finally done! Or is it? Alas, our goal is not to deliver the results–it is to make sure our audience actually uses them.

As we consider the audience’s needs, bear in mind learnings from the education and training sectors. Sharing MR results is essentially training people to apply new information. What lessons from these other sectors can we apply?

• Repetition is important. Just like when we’re teaching kids how to spell, practice makes perfect. The first time people hear information it has a certain amount of impact, but if they hear that information two or three times it greatly improves retention.
• Multiple modes are important. We’re more likely to retain information if we read it, hear it, and see it in visual displays like graphs and charts. Better still, if we get to apply it through practice quizzes, role-playing, or other interactive exercise. In contrast, if we only read something, we retain far less.

What does this mean? It means we can’t just email out a research report and assume the project has now been delivered. If we expect people to retain and apply research results, we have to create a more comprehensive delivery strategy. Presentations, email follow-ups, executive endorsements, podcasts, custom follow-ups, 1:1 briefings, posters, and internal blogs are some of the options that can be combined to make sure we reach our audience multiple times using multiple modes. Lather, rinse, repeat.

[Want to learn more about delivering a market research project’s results? Check out our 34-minute, flash-based class “How to Package and Deliver Market Research Results for Maximum Impact” available online now.]

If you throw “social media” and “market research” into a blender, what do you get? It depends on the exact recipe you select. The good news is that whatever you decide to try first, you can start with some free samples.

In terms of what many organizations can most easily test today, social media monitoring, social sampling and prediction markets are great options.

If you want to learn more about these methods, their pros and cons, and suggestions for getting started with them, please check out Research Rockstar’s new class: Social Media Meets Market Research.

Sign up for the class before May 25^th with coupon code 472Z1 and the class will be just $99.

This 73-minute online, self-paced class includes an automated, interactive self-quiz. With this fast, easy market research training, you will be ready to start “tasting” the samples, without the risk of a nasty BassOMatic mess.

Please check out the class preview: PREVIEW LINK.

Is your organization doing more market research?  Are people outside of the market research department becoming increasingly involved in research, either as “do it yourself” researchers or as members of cross-functional project teams?  Imagine a baseball team made up of three professional ball players and six middle-aged guys from the local coffee shop. Wouldn’t those well-intentioned coffee drinkers need some practice before that first game? Well if you want your whole research team to play at their best, here are a few things to keep in mind:

Selecting Market Research Training Topics

When selecting topics for a MR training program, there are three key questions to consider about your audience.

• Do they need to know how to be informed market research clients? Market research clients can contribute more meaningfully to the process and will have more realistic expectations about their role if they’ve received some basic training. If you have cross-functional teams contributing to research projects, this becomes even more important as individuals from marketing, manufacturing, product development, and sales will have different market research knowledge and expectations.

• Do they need to know the basics of how to run small projects start to finish? If people outside the market research department are doing online surveys, Facebook polls, or other small scale research projects from start to finish, it’s useful to have them trained on questionnaire design, project management, basic data analysis, and data reporting.

• Do they need an understanding of specific market research topics? There are a lot of technical terms and best practices in market research, and educating people upfront pays dividends later. You’ll spend less time correcting bad assumptions and you’ll minimize dead weight. Suppose you’ve got some significant research coming up, say a market segmentation study or a new customer loyalty tracker.  This would be a great time to do some training on those topics.  There’s a lot of good information that can be covered in a class on specific market research project types that would bring everyone up to a basic level of understanding before you launch that big new project.

Let’s Play Ball!

By providing some basic training in market research, you’ll not only reduce the overall risk and improve the success of market research projects, but you’ll also encourage valuable contributions from non‑researchers. Instead of yelling at the ref from the sidelines, they’ll be in the game — and that’s what makes for a good season!

For information on specific Research Rockstar classes click here. To learn more about Research Rockstar’s onsite training click here.

We get a lot of questions from people evaluating low-cost online survey platforms. And given that there are now more than 50 options available (see partial list at the end of this article), it is a bit confusing. Before you start evaluating options, consider these checklist items:

• Do you want a platform that integrates with a third party for panel access, or do you plan to provide your own sample?
• Do you need support for constant sum, rating, and semantic differential scales?
• Do you want to be able to impose complex quota rules?
• Do you want to be able to set up a library of commonly used questions or survey designs?
• Do you prefer simple questionnaire designs (perhaps for colleagues who may find more advanced options confusing)?
• Do you plan to do data analysis in the platform itself, or will you be exporting to SPSS or other programs?
• Do you want to customize and automate delivery of reports to colleagues?

Depending on your answers, your platform needs will vary. In fact, a “yes” answer to 2 or 3 of these will narrow your field of options down considerably.

Here are some of your lower-cost online survey platform options:

AYTM – http:/askyourtargetmarket.com/

Checkbox – http:/www.checkbox.com/

Cvent – http://www.cvent.com/

QuestionPro – http:/www.questionpro.com/

SurveyGizmo – http://www.surveygizmo.com/

SurveyMonkey – http:/www.surveymonkey.com/

Wufoo – http:/wufoo.com/ (just acquired by SurveyMonkey)

Zoomerang – http:/www.zoomerang.com/

And many, many more!

[If you haven’t done so already, click HERE to sign up for Research Rockstar’s Market Research Newsletter.]

Imagine a six year old who truly believes Santa grants all wishes. Even though he lives in a sixth-floor Chicago walkup, this child firmly believes Santa will come down the (nonexistent) chimney and leave a pony under the Christmas tree. Imagine the tears on Christmas morning when there’s no pony.

Most of us have experienced the feeling of being oversold, whether in business or in our personal lives. It’s painful. And it’s the last thing you want your internal colleagues, team members, or other associates to feel at the end of a market research project. Unfortunately, it happens. Maybe the results were unexpected, maybe the scope or methodology ended up being somewhat different than what they’d pictured. For whatever reason, the people who should have joyfully embraced your research results are unhappy. And after you’ve invested weeks or months of effort, time, and money, unhappy internal clients are the last thing you want.

How can we avoid this outcome? Be careful not to oversell market research. Those of us who do market research professionally tend to get enthusiastic. We are typically people who enjoy designing and implementing research methodologies, who like to dive into mounds of data and extract meaningful results. In our enthusiasm for research, we have to be careful not to over-promise. Realistic expectations are the key to satisfied clients, especially for riskier types of projects.

Product Concept Testing Example: What We Can and Cannot Promise

Product concept testing can be designed to help identify product ideas or feature combinations that have the highest potential share of the market. Key word: potential.

But there are challenges with product concept testing research. Sometimes people can’t respond to particular product concepts that are too new or innovative. They don’t have a frame of reference, so it is just too hypothetical. Henry Ford once said, “If I’d asked people what they wanted, they would have said a faster horse.”

Even in more mature categories, product concept testing research has limits. After all, a lot of research depends on asking people about their attitudes and behaviors, and even well-intentioned participants cannot report such information perfectly.

What can we promise about product concept testing that is realistic? We could design the project and promise we will:

• prioritize features
• prioritize among multiple new concepts
• identify likely tradeoffs between features and price
• estimate a demand elasticity curve
• weed out blatantly bad ideas
• identify potential demand deterrents or sales objections to a particular product concept

We also need to let colleagues know that there are many factors beyond our control. A major competitor may come out with a new product that influences purchase criteria. A new entrant may come out with a big splash and disrupt brand preferences. A shift in the economic environment could change price sensitivities.

Product concept testing research is still very worthwhile (and far better than not doing any at all), but it is an example of the type of project that has inherent limitations. It’s always better to do some research than none, but we don’t want to over-sell it.

Bottom line

If you’re doing market research on a project that is known to be risky in terms of how well it can predict actual customer attitudes and behaviors, you have two key steps to take:

1. Consider augmenting survey research with other methodologies.
   Social media monitoring? A prediction market? Ethnography? Rapid prototype testing?
2. Set realistic expectations. First, be certain that internal clients have realistic expectations about what they can and can’t expect. Let them know that inconclusive, contrary, or contradictory results can come up. If they do, it may require further research or the use of other methods—all of these things are part of the real world.

With realistic expectations, people will have the right mindset. We won’t have to worry about disappointing them in the end, and that saves everybody a lot of aggravation.

So don’t even try to be an all-powerful Santa. If that pony won’t be trotting out from under the tree, make it clear in advance. That city child would be happier with a shiny new bicycle anyway.

For more on Social Media Research, sign up for Research Rockstar’s White Paper on the subject.

Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP

Even if a market research project produces a pile of perfect data, we still face the fundamental challenge of analysis — making sure that we’re analyzing the results comprehensively and objectively. In other words, without bias.

Let’s say you’ve done an online survey. You identified your objectives, thought carefully about sampling, and designed a great questionnaire. You monitored data collection and carefully cleaned your dataset. Even after all this painstaking work, risk still exists. You still have to analyze the data, and it’s here that unexpected errors often creep in.

WHAT WE SEE 

Human beings have a natural tendency to look for and see things that confirm our hypotheses, or that are consistent with our personal experiences. The result? It’s as though someone has applied a highlighter to the data so that the things we expect or believe jump out at us.

WHAT WE DON’T SEE 

We tend to overlook things we don’t expect or don’t believe in, and have an unconscious desire to ignore things that are inconsistent with our experience and expectations. Amid the flood of tables, charts and graphs produced by our survey, there may well be unexpected results that are more significant and useful than anything we anticipated, but because we’re not looking for them, there’s a good chance we won’t see them. If you’re a doubter, take a look at this video.

ATTRIBUTES OF A GREAT RESEARCHER

Some people are truly gifted at market research analysis. Some key signs?

• They’re able to challenge their own assumptions
• They’re willing to play devil’s advocate and challenge the way other people are looking at the data too.
• They recognize unexpected themes that sometimes appear in a data set.
• They resist the temptation to embrace the first story the data reveals; they’ll look for multiple stories so that they can determine which ones have truly compelling value.
• They can deliver bad news, such as “You know, your baby is ugly”; meaning that your favorite project, concept, or idea is getting negative results.

The ability to objectively and thoroughly look at data to see unexpected patterns is the key to being a great market research analyst. Even when a research project is perfectly designed and executed, there’s a real risk that it will fall apart at the analysis stage, wasting time, money and an opportunity to profit from hard work.

Market Research Bias Problems

In any analysis project, researchers need to sanity check for the following challenges:

1. Positive Bias: The unconscious tendency to see what we’re searching for or expecting to find.
2. Inattentional Blindness: An unconscious tendency to miss what we’re not looking for or don’t expect.
3. Happy News Bias: The inability to acknowledge that there is some “bad” news.

If these problems are at all evident, consider the following options:

1. Challenge Assumptions: If you see results that closely match your expectations or confirm what you already know a little too neatly, challenge yourself. Embrace the discipline of being objective, or hire people who can be.
2. Develop a checklist: Develop a personal checklist to help you maintain objectivity. With experience, you’ll discover where your own biases tend to surface. A checklist of those biases can help you avoid objectivity traps.
3. Hire those with the skills you lack: Someone on the analysis team has to be able to see what the others miss, and be willing to speak up about it. Hire people who see things differently, and create an environment where they are willing to be vocal about what they see.
4. Diversify Analysts: Different people have different biases and will have different interpretations of a data set. With a team, it’s possible that you’ll cancel out each others’ biases. We each have our weaknesses and blind spots, and a team approach will help to mitigate them.

Market Research Analysis: Did You Catch a Fish or a Boot?

Embarking on a marketing research project is a lot like fishing. You can pick the day, the time, the spot, the lure and the line, but you can’t control what you hook. Before you head back to the dock, take a good look in your hand and see if what you’re holding is a five-pound trout — or an old boot.

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Imagine it’s Friday evening. You’ve been scrambling all week, and you’ve decided to unwind with a nice dinner out. The fellow at the desk next to yours has been raving about this new bistro in town—the best steak, perfect wine, and dessert to die for. So you decide to treat yourself, and…

The hostess is rude, the salad limp, the wine warm and the steak unrecognizable. What are the chances you’ll be going back there again? How likely are you to take future restaurant tips from the guy who set you up? Might you even tell a few friends about your horrid experience?

Yes, this has something to do with market research. Or, to be precise, surveys. When bad surveys are circulated, the company that sent them out becomes less trusted. The “consumer” becomes an unhappy customer, and may even tell others about their bad experience—with surveys in general or with the specific company.

WRITING GREAT SURVEYS IS IMPORTANT TO EVERYONE

Anybody who’s had a bad survey experience is likely to have a tainted perception of the process, and that can come out in a number of ways:

1. Participation: They may be less likely to take surveys in the future.
2. Attitude: They might spread negative word of mouth about the company that sent the survey or about market research as a whole.
3. Skepticism: They may be skeptical the next time they see market research results.
4. Investment. If they are business professionals, they may be less supportive of their organization’s investments in market research because they just don’t trust the process.

BEING A GOOD CITIZEN

As good citizens of the market research community, we have an obligation to make sure the questionnaires our organizations are distributing are impeccable. Even those coming from the well-intentioned but usually untrained DIYers. The challenge for many managers is the huge number of colleagues now using low-cost tools for creating surveys. An opportunity, yes. But without proper training and oversight, the chance of creating a bad survey is greater than ever. Here are some low-cost options to help avoid that:

• Examples – Provide a template library of commonly used, approved questions. Demographic questions at minimum, so that your colleagues will be collecting consistent profiling information but avoiding questions that may be too intrusive, or too vague to be useful.
• References – There are some great books out there. The “Handbook of Marketing Scales” by Bearden, Netemeyer and Haws (Sage Publications, 2011) is one favorite. A little technical, but absolutely readable to anyone willing to make an effort.
• Quality Assurance – Appoint one or two people as the Survey Q&A Reviewers, and give them the responsibility (and authority) of sanity-checking any surveys before going live (especially those being sent to your valued customers). Make sure this role is publicized and endorsed by managers.
• Keep it simple – There are a lot of tools out there, and while it might seem counterintuitive, sometimes you’re better off limiting the choices. Instead of many different question types, limit it to a handful (say, multiple and single choice, and Likert scales) to keep the surveys manageable, and therefore less prone to design abuse.
• Training – Basic skills are important. Start with new employee orientation materials and train your workforce. I’m a little biased here since Research Rockstar is an online training company, but there are also others that offer seminars and webinars, including the MRA, Burke Institute and ESOMAR.

SURVEY QUALITY IS CRITICAL

Survey quality is important to those who create them, those who take them and those who depend on their results. It’s in everyone’s best interest to assure that what the public sees reflects the quality and professionalism of the market research industry. As I’ve mentioned in other blogs, having a few good policies is a great place to start. And it might just earn you a nice dinner out come Friday, too.

[Planning to write a questionnaire? Check out a preview of Research Rockstar's questionnaire design process class.]

Typically when we think about market research surveys, we think of questionnaires that have 20, 30, or even more questions. Getting qualified people to complete these questionnaires has become a serious challenge.  One alternative is the single-question poll. After all, you’re much more likely to get high response and low dropout rates if you can simply say, “Hi, we have a single question we’d like your opinion on”, rather than requesting a novel’s worth of responses.

Facebook is making polling insanely easy these days, and several polling applications are available on Facebook. Creating a single-question poll is a snap, and then you can make it available on your fan page or your personal page, or you might invite friends to take it.

Let’s say your company has a fan page with hundreds or thousands of fans. You can simply post the poll on the page. No fan page? Facebook also gives you the option to “purchase sample.”  Only want men and women from the U.S., or only interested in men from Mexico for your particular poll? No problem. While gender and country are currently the only 2 options offered, I wouldn’t be surprised if Facebook came out with more powerful select options soon.

Facebook Polls: The Good News

1. Speed. A key advantage of Facebook polls, clearly, is speed. Results may be available within an hour.
2. Numbers. With over 600,000,000 users as of early 2011 (compare that to the US population of 310,000,000), you’ve got a lot of potential respondents.
3. Cost. Having done some experimenting with Facebook polls, I can tell you it’s very affordable and perfectly appropriate for certain types of topics. Placing a poll on Facebook is free as of March 2011 (though of course that’s subject to change), and their current sample pricing is very low.

Facebook Poll Limitations

1. Limited selects. Currently, you can choose from just gender and country (though again, I am sure this will change soon given how many options they offer for selecting audience members for their ads).
2. It’s “only” Facebook. Obviously, not everybody is on Facebook. While it does represent a broad mix of ages and countries and has a pretty even gender mix, it’s likely not the best match for specific groups such as business decision makers. My experience so far is that it’s good for consumer-type topics more than business-to-business polling.  But if you’re running a business-to-business company and you’ve got a fan page, it’s worth testing.

The Challenges of Polls in General

1. One Question. It’s a single question, so you have to craft that question carefully and understand who’s responding, keeping in mind that there’s a lot you don’t know about those respondents.
2. Polls don’t represent everyone. Some skeptics would say that the people who opt into these types of polls may not be representative of the broader population. Seems to me a single-question poll has a higher probability of broad response than a longer survey, so perhaps it balances out.
3. It may raise more questions. When you look at the results of a single-question poll it can raise more questions. You don’t have the benefit of a logical branching or skip pattern to follow up on specific answers. You have little context. Imagine a scenario where we ask participants to select which of a list of 5 features is most important when buying a tablet device. Say the poll finds that one item markedly stands out. On one hand that’s great, but on the other it raises the question of “why”? And how might that have varied by customer type, etc.? Cool data, but it leaves us begging for more.
4. Limited Uses. A single question can give you directional data, maybe even help you uncover some interesting things worth further investigation. But you aren’t going to make a million-dollar decision based on such data.

A Simple Test

I did a test the other day for $15.00.  I selected ‘men and women from the U.S.’ and had 50 responses within two hours (that’s 30 cents per response). Granted I don’t know much about these people, and I asked a pretty generic question, but it was very fast and affordable.  For topics where some data is better than no data, that can be totally appropriate.

Polls Are a Viable Option for Fast, Directional Data

Given the caveats that we don’t know much about the respondents and we don’t know enough to make extrapolations, there’s nothing wrong with asking a quick question of the Facebook population. Not every project warrants a big budget or weeks of effort. And with 600,000,000 users, there is a huge sample source just clicks away.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]

Do you cringe when you hear the word “policies”? Most people do. After all, policies often mean bureaucracy.  But in the case of market research, clear policies will minimize the risk of data quality headaches, customer over-surveying, ethical breaches and more.

Indeed, a thoughtful, well-communicated set of policies is more critical today than ever before, with so many people conducting ad hoc or “DIY” research. Well-intentioned individuals often make mistakes that could be avoided through awareness of a simple set of company-wide market research policies. Even organizations with central market research departments find it challenging to control “rogue” research—but promoting a set of policies will help minimize the risks.

Below are examples of market research policies that will promote basic, best practices:

1. Frequency. Over-surveying can lead to customer frustration and ultimately, poor response rates. Thus, a key policy is to specify how many times a year a single customer can be invited to participate in research.  Three times? Five times? There is no “right” answer for all organizations—it varies by customer type. But a rule should be in place. In this way, employees can avoid inundating customers with volumes of survey requests.  Of course, this also requires having a mechanism in place to track this.
2. Quality. All direct communications coming from your company are indicators of your brand’s quality, and surveys are no exception.  You must ensure that a kind of “quality control” resource exists to ensure that nothing sub-par gets released.  This job includes checking grammar and questioning content and logic.  For example, one common complaint about colleagues who do ad hoc research is that they may ask too many intrusive questions (a big turn off for customers). This resource could be a person, a team, or a defined process.
3. Permissibility. The best way to prevent unsanctioned surveys is to make sure everyone knows how to request and get approval for market research projects.  Your company can specify what types of research must be done through central market research (if it has such a department) and what can be done by other functional areas.  A simple research request process should be in place so that employees can submit a standard form that can be used to trigger an assessment and approval process.  Too onerous? Then how about a simple policy stating, “Any surveys over 10 minutes in duration must be approved by the central market research (or if none exists, marketing) department– no exceptions.”
4. Methods. Company guidelines should state policies for both qualitative and quantitative methods. For example, “All online surveys must be fewer than 30 questions.” Or, “Recruiting customers for in-depth interviews must be coordinated with the VP of sales at least two weeks ahead of time.”  These are just two simple examples, but you get the idea.
5. Incentives. An incentive policy should include guidelines for types of incentives and under what circumstances they can be given out.  Inform your employees ahead of time about whether or not your company restricts cash incentives or any type of “gifts” to customers.
6. Solicitation. A strict non-solicitation policy must be in place. Selling “under the guise of research” is entirely unethical and must be avoided. Even the appearance of solicitation can lead to big problems for your company. Surveys must not be used as thinly veiled lead generation mechanisms. [Click HERE to get more tips on survey design.]
7. Confidentiality. A confidentiality policy will ensure your employees understand how to use research information responsibly and will show your clients that you value their privacy.  Obviously, it is essential that confidential information is protected, so train people on what information is confidential, how it should be stored, and how it should be treated (internally and externally). Another realm of confidentiality lies in what company information is shared in a research study.  Consider rules that will avoid unwanted leaks. For example, a policy may be that any research related to new product concepts must be approved by the VP of marketing.

Market Research Training Via Policies

While these simple policies may appear obvious to an experienced researcher, it is important to present them to all research-related colleagues. Include policies in employee orientation materials and provide reference materials for all employees who may in any way touch market research—whether it’s the DIY kind or not. Just by raising awareness that there are policies, you will be providing subtle training on best practices.

[Do you have staff that could use some market research training? Check out our online classes; most are under an hour, and all can be viewed conveniently from any web browser.]

Did you learn everything you possibly could from your last project—not just from the final results, but by examining the process itself? Conducting a detailed review, a post mortem so to speak, can help pinpoint exactly what worked—and what needs work.

For example, let’s say a recent project used in-person focus groups. Did you and your team discuss how the process went?  What worked well about recruiting and what didn’t? Did the facilities and the moderator meet your expectations? Did the discussion guide support the intended goals? All great questions to reflect upon.

So, how do you conduct a project post mortem? By objectively evaluating the process, step-by-step. But remember, the goal is not to place blame—it’s to find opportunities for improvement. So hone your scalpels and focus your microscopes; come time for your next project, you’ll be glad you did.

Let’s use the case of a quantitative project and break it down, stage by stage. For each stage, I suggest a few key questions to get you started.

Stage 1: PROJECT DESIGN

Did the specification of project scope and objectives go smoothly? Was it challenging to get all team members to agree on a reasonable, finite set of project goals? If an outside agency was engaged, how did the process of writing an RFP and selecting and hiring the agency go?

Stage 2: QUESTIONNAIRE DESIGN

How many iterations were anticipated and how many were made?  How did the programming and pretest go?  Were things discovered in the pretest results that needed significant reworking?  If there was a lot of reworking late in the process, could it have been avoided by preliminary qualitative research or by clearer thinking about project hypotheses?

Stage 3: SAMPLE SELECTION AND SOURCING

Did the sample sources you planned on using work out?  Were you satisfied with response and incidence rates?  What percent of records from a sample source did you have to toss because they were suspicious?

Stage 4: DATA ANALYSIS

Was a clear data analysis plan in place so that everybody knew exactly what was going to be done and how?  Were there data analysis errors?  Was there a lot of back and forth asking for more cuts of data, which could have been avoided with better planning?

Stage 5: DELIVERABLES

Were deliverables created on time and error free? Were reports professional looking and well received?  Were there other types of deliverables that internal clients or colleagues would have preferred?

And perhaps the most important measure of success: did people embrace the results, leading to important conversations, decisions or strategies?

WRAP IT UP: THE REPORT CARD

Boil down takeaways into a single page summary, and keep it factual. Avoid sneak attacks. Nobody should dread report card time. You’re simply identifying improvements for next time around. This is your post mortem deliverable, an at-a-glance reference for next time.

Here are some options:

Self-assessment Report Card. In this case, the project team grades the project by phase. A weighted scorecard approach can be used by considering grades for items such as budget adherence, timeline adherence, usefulness of final results, and so on.

Client feedback. Collect client feedback (from internal or external clients). Ask for feedback on overall satisfaction and on specific projects’ aspects such as communications, schedule, and deliverables. How well did the project results and deliverables meet expectations?  Did the results sufficiently support intended decisions and strategies? If not, why not? Giving clients a chance to grade the project can uncover some surprising areas for improvement; some clients won’t complain along the way, but sure do have feedback to give when asked.  Get it promptly, and address it, so that they won’t turn into market research skeptics.

Whatever approach you take, formal or informal, self-assessed or client feedback-based, you will certainly find opportunities for improvement. The key? Capture post mortem information before rigor mortis sets in, within a week of the project’s conclusion or sooner.  Projects over three months long may call for a midway mini-review.

[If you haven’t done so already, click HERE to sign up for Research Rockstar’s Market Research Newsletter. ]