Numerous articles have been written, and debates engaged, about the question, “Are market researchers bad marketers?”  It does strike one as odd—that a profession so driven to understand customer attitudes and behaviors, can’t seem to apply the discipline for its own marketing—and ultimately revenue-generating—benefit.

Three great articles have poked at this issue this past year, by Edward Appleton, Ron Sellers, and Dana Stanley.

Is it just the case of the cobbler’s children having no shoes?

Well, I got a big clue earlier this month.  I won’t name names, but I heard a researcher from a well-known firm talk about their, “new approach to market segmentation.”  And about ten minutes in, I reached three conclusions.

• Yes, his company is doing some cool stuff to leverage social media research to gain insights about specific target markets
• But he was using the term “segmentation” incorrectly; his methodology is about profiling existing, known segments previously identified by his clients—not discovering the best ways to segment a market. What his firm is doing is profiling, or creating personas as some prefer, but it is not a “segmentation study.”
• And apparently his clients don’t care.  His firm is apparently doing very well selling segmentation studies that rely on “listening” to social media—not asking questions. And on profiling segments; not defining them.

I have to say, this speaker deserves major kudos for understanding his market segment; that there are a lot of marketing decision makers who want fast, social media-based profiling of their existing segments. This guy understands what his market wants, and he is giving it to them.

So what did this all teach me?

Often, making a complex thing really simple is what leads to marketing success.  A classic example: for years AOL was the preferred email platform of choice for the average American consumer; we techie types cringed, but the masses loved it.  And unfortunately, the masses don’t always appreciate the brilliance of more complex products—or in our case, of more rigorous market research.  Calling a profiling study “market segmentation” may make researchers cringe, but to a lot of people who buy market research, it may be just fine.

[Want to learn more about market segmentation or other market research topics? Click here to request Research Rockstar's class catalog today! Now, available in print and by download.]

Sadly, I recently encountered a Google Consumers Survey question that proved even short questions can go badly awry.

Respondents were asked “Will the $95 IRS penalty motivate you to shop this October for an Obamacare health plan?” and could choose “Yes,” “No,” and “Not Certain.”

Every day I quickly review about a dozen recently published surveys to choose a few to write about on Researchscape.com, setting aside ones like this with obvious errors. Good questions are hard to write, and I’ve written my share of bad ones. This question has quite a few problems:

1. The “$95 IRS penalty” is incomplete – The penalty is $95 per individual or 1% of household income, whichever is greater. By understating the penalty, this most likely depressed “Yes” votes.
2. The “in October” is overly specific – Some may purchase insurance in advance of October to avoid the penalty. This probably depressed “Yes” votes.
3. Obamacare is a loaded term – Originally coined as a pejorative by Republicans opposed to the Affordable Care Act, it has only gradually come to be used by Democrats. I’ve used Affordable Care Act in healthcare surveys I’ve conducted for precisely this reason. I’m not sure of the effect this would have on the overall results; it might increase nonresponse, for instance.
4. “Obamacare health plan” is too narrow – The intent was to find out if people would buy insurance because of the penalty, but they do not have to buy insurance from ACA health exchanges, as this wording may have implied. This also most likely depressed the “Yes” votes.
5. The sample is wrong – The question is asked of everyone, when it really should be asked of those currently without health insurance with a different question with new wording for those who currently have insurance. The user should have added a screening question with Google Consumer Surveys’ “Pick audience” functionality.

Whew, that’s a lot of mistakes for 16 words and 88 characters!

Questionnaire writing can be incredibly demanding, even when that questionnaire is only 16 words long.

Speaking of healthcare, maybe more Do It Yourself researchers should have their questionnaire’s vitals checked out by SurveyMedic?!

[This Guest Post was written by Jeffrey Henning of Researchscape. Be sure to follow him on Twitter at @JHenning]

It’s a bit of dogma often claimed by market researchers, though it is circulated in other fields as well (including software development).

Why is it so popular? Primarily because it does pass the gut test: market research projects that are fast and good are unlikely to be cheap. Ones that are cheap and fast are unlikely to be good, and so on.

However, the dogma has become outdated. Thanks to new technologies and methodologies, market research projects are being completed faster and less expensively than they once were—and are still meeting quality needs. Or to employ yet another over-used aphorism, we have learned to do more, with less.

Of course, we still make trade-off’s in designing and executing market research studies. Just different ones.

Today, the balancing act is no longer about cheap, fast or good. Today it is about projectable, innovative or simple.

We often want research that is projectable, such that it accurately describes a target market or population of interest.

We also often want research that is simple, such that its risks can be managed and it can be delivered according to intended budget and schedule parameters.

And these days, we are also often seeking innovation. Not for the sake of innovation itself, but for the sake of overcoming known issues such as the limitations of self-reported behavior and the need for more accurate emotional measures.

Yet these are trade-offs.  Cool new methods and tools that help us gain rich projectable insights into emotions aren’t simple.  Market research projects that are simple and innovative are rarely projectable. And based on my experience, those that are simple and projectable are rarely innovative.

What do you think: Is the current trade-off between projectable, innovative or simple? And if so, can you think of a project you have done that was, in fact, all three? Is it possible?

[Get the best market research articles of 2012 in our convenient compendium. Topics include Facebook for research, Google consumer surveys, mobile research and more.]

In catching up on market research reading, we stumbled on this little gem from Question Science Blogspot.  In this article, Jon Puleston tells us about some surprising statistics he overheard while attending the ESOMAR 3D conference at the end of 2012:

350 out of 36,000—Porsche culled through 36,000 social media responses and found that only 350 were “useful”. Significantly, all of the comments were processed manually. This suggests that deciphering data from social media could be a poor investment.  So, can text analytics software accurately decipher social media comments, and are the comments even worth deciphering?   Clearly, this is going to vary by topic, brand in question and scope. Some brands/keywords get a lot more “garbage” than others.  What we have found here at Research Rockstar is that you have to do some serious testing of your topic/brand name/keywords of interest before you invest significantly in social media analysis.

240 hours– The amount of time spent by a market research firm analyzing text from 1000 Facebook users.

.18—A survey by Jannie Hofmyer and Alice Louw from market research company TNS, showed a surprising lack of correlation between “aided awareness of a brand & purchase activity”. Their research revealed that surveys are routinely constructed incorrectly and contain questions that are incapable of measuring behavior. Customers and non-customers of products should take different surveys to create relevant survey data results.

50%– Peit Hein van Dam, from digital tracking company Wakoopa, tracked a 50% variation between the claimed readership level of a Dutch newspaper and the readership level tracked on mobile devices and computers. “Cookie” tracking proves to be largely inaccurate in counting unique visitors and web traffic.

[Want access to more market research articles and training materials?

Sign up for the Research Rockstar newsletter: SIGNUP]

This is a questions I get frequently from Research Rockstar students and other people I run into—even just recently at the MRA’s Joint Chapter conference in Miami.

Who has time or money to attend two or more major events a year—if that? Other than companies with booths and selling things at the events, not many.

Both the AMA and MRA offer great annual events, and additional conferences throughout the year.

The comparison I most often get asked about is AMA versus MRA annual conferences. I will share my experiences with these events, and hopefully this will be helpful.

At the annual AMA market research conference, I meet a mix of market research and marketing professionals. Personally, I like that mix—it leads to interesting discussions and prevents us from treating market research as a silo. The marketing people tend to be those who have roles that mix market research and broader marketing responsibilities.  The market research attendees are a good mix of client-side and supplier-side researchers.  The presentations at this conference tend to showcase a lot of case studies, and cover a mix of market research standard methods and emerging solutions.

Bottom line: for great networking with a diverse crowd and broad learning, the AMA conference is an excellent choice.

At the national/annual MRA conference, I meet a variety of market research professionals; client-side researchers, suppliers, and even some educators.  In the past, these events felt a bit too skewed to the supplier side, but that has changed a lot over the past three years.  The event now gets a nice representation of client-side researchers—people who can talk firsthand about the experience of buying, managing and delivering in-house research.  To me, this is important: without a mix of attendee types (clients and suppliers), the networking just gets too dull.  While different than the annual MRA conference, even at last week’s joint chapter conference in Miami, I saw great presentations and met several people from client-side market search departments.

The annual MRA event is also great for a mix of case study presentations (showing actual market research applications) and in-depth learning related to new methods.

Bottom line: for great networking and in-depth learning, the MRA conference is an excellent choice.

[Update: it appears that the AMA may be discontinuing its annual market research conference? I will update as I learn more!]

[Want to receive market research news and articles by email? Click here to sign up for Research Rockstar's newsletter.]

Well, curiously enough, there are similar trade-offs when it comes to market research survey design, and, in our case, “resolution” is defined by the number of points we choose for our scaled questions.

More Points Equals Higher Survey Research Resolution

How many points in a scaled question is “enough”? That’s a pretty common question in survey design. If you’ve spent some time with online market research surveys, you’ve probably noticed that they often have five points.  “Please rate your satisfaction on a scale from 1 to 5, with one representing ‘not at all satisfied’ and five representing ‘extremely satisfied’.” Sometimes, though, you may see a 7-point scale instead of 5. Is 7 points an improvement? How about 11? Is higher resolution better? The answer is, “that depends”.

Seeking Balance in Survey Scales

Ask this question of two researchers and you’ll likely get two very different opinions, but ideally we want to walk the line between simplicity for our survey takers and accuracy in our data. If we expect to see a lot of variability in the responses, higher resolution will allow our respondents to be more precise, providing better data. As an example, we tend to use larger scales for surveys about customer satisfaction because of the wide variation in customer experiences with a given brand.

Even or Odd Scales?

The choice of either an even or odd point scale really boils down to one issue: Do you want to give your respondents a “neutral” option? Market researchers tend toward odd scales for most projects, so 5 and 7-point scales are common, and even 11-point scales are seen occasionally.

There’s a debate, and it goes like this:

• Even forces a choice.  There’s no riding the fence with even scales — your respondent has to come down at least somewhat on one side of your question or the other. My personal take is that neutral can be a valid opinion and I don’t want to force people to say something that’s not true. And in some cases, people are too nice, so they might answer on the positive side — even if they don’t feel that way — unless you give them the option of neutral. It’s something of a quandary, and it requires some judgment on the researcher’s part.  Familiarity with the target population in question helps.

• Odd allows an easy-out.  Well, maybe. Some researchers believe that people will default to neutral because it’s easier, but there’s a debate about whether that really happens. In my experience, it is not an issue.  It used to be a problem years ago when many surveys were done on paper—we would call it “pencil whipping” (respondents would “whip” their pencils down the middle column of a survey grid).  But for online surveys in particular, I see no evidence that this is a notable issue.

Plan for Future Surveys

If you intend to be doing these surveys for a while, it’s worth thinking about picking a scale and sticking with it.  Even or odd, 4 or 7 points, standardize your system so that you have some consistency across your market research surveys over time. Some day you’re going to want to compare results, and having a standard format will make your life a little easier.

When in Doubt, Go Big

If you still can’t decide on a scale to stick with, err on the side of higher resolution. Just like in photographs, you can always collapse the scale when you’re doing your data analysis, but you can never expand it (increase the resolution) once you’ve captured the data.

Know Your Survey Goals

The choices really aren’t that complex. Larger scales allow a greater degree of potential accuracy, but require a bit more thinking by participants. As a researcher, you need to make these trade-offs carefully, so that your research delivers the best picture possible.

[Want to learn how to use questionnaires to capture attitudes and behaviors while avoiding the common mistake of over-relying on 5-point scaled questions? Check out our live, instructor-led class, Ask It Right: Choosing Scales & Answer Options for Online Surveys.]

It certainly can be.

But I can also tell you, after 25 years of doing market research, that there are known strategies for managing project-related stress.  My favorite is one Research Rockstar students hear me talk about frequently: the preemptive strike.

Most stress-inducers are due to unmanaged risk factors. Things that can go wrong in a market research project that cause budgets to be squeezed, timelines to slip or success criteria to be unmet.  Trying to manage these risks after they have already emerged is often difficult and highly stress-inducing.

Example of Market Research Project Stress

Consider the market researcher who has committed to a project where the screening criteria are known to be hard. Sure enough, at some point, it gets stressful.  Quotas aren’t being met, deadlines are looming, and the client (internal or external) is becoming unhappy. It is a brutal time, and stress levels soar.

In this scenario and others, the best option would have been a preemptive strike. That is, taking steps to make sure the problem never got to the stress-inducing point. In this example, there are 4 ways to do this:

1. Getting clients to pre-approve fallback strategies (what screening items could possibly be relaxed in the event it became necessary).
2. Having additional sample sources (and budget) on stand-by if needed.
3. Educating the client on the possibility that some quotas may not be met and why.
4. Having secondary sources that prove why some of the screeners are likely to be difficult (so that you can demonstrate that the possible data collection issue would not be due to your ineffectiveness, but to size-of-universe realities).

Market Research & the Preemptive Strike

Luckily, most market research risk factors that cause stress are known, and best practices exist to manage them. In many cases, the best strategy is to employ a preemptive strike as a way to neutralize risks before they blow up your project.

[Want to learn other ways to minimize risks and maximize efficiency in your market research projects? Click here to check out our Market Research Project Management 4-week Power Program!!]

Less is More, for Market Research Credibility

In the February 2012 issue of Inc. Magazine, Twitter and Blogger co-founder Evan Williams promotes the idea of doing less. That is,  “If you have too many things to think about, you’ll get to the superficial solution—not the brilliant one.”  For we researchers, this is a hard balance. On one hand, we know that the value of analysis seldom comes from focusing on one or two data points—it comes from identifying recurring themes and patterns. Indeed, we often talk about “weaving” together a story from multiple data points.  But we also know that at the other extreme, dumping too much data in a client’s lap, leads to disaster: they turn off, stop listening, and even judge us as unable to prioritize or synthesize, thus hurting our profession’s credibility.  Allowing people to focus more on fewer items, does enhance how market research is perceived.

Less is More, for Richer Market Research Analysis

In the Sunday New York Times (January 20^th, 2013 edition), Matthew E. May wrote about, “The Art of Adding Through Taking Away.”  The article points to the strength of this wisdom through ancient proverb and more recently by quoting Jim Collins, who apparently observed that, “A great piece of art is composed not just of what is in the final piece, but equally important, what is not. It is the discipline to discard what does not fit — to cut out what might have already cost days or even years of effort …and marks the ideal piece of work, be it a symphony, a novel, a painting…”

This is so true for those of us who write market research reports. It is always a challenge to hold back—we find so many interesting and tempting data points in a single study. Yet we know that the discipline to reduce our work to its core essence is essential, and will even help us to create more meaningful analyses. A good market researcher will find many interesting things to report; a great one will focus on fewer items but bridge the gap to actionable insights.  The restraint is not easy, but is always rewarded.

[Report writing strategies are covered in Research Rockstar’s Project Management class. Next session starts February 28^th and meets once a week for 4 weeks.]

While running a market research agency years ago, I learned the importance of specialization the hard way.  Sometimes it was better to take somebody who was a market research project manager and have them just focus on statistics—if that’s really where their talent lay.  Over time, we actually restructured the organization this way—a Copernican revolution, if you will, that placed human talent at the center of consideration. We had a survey programming specialist, a data collection specialist, an editing specialist, a graphics specialist, and so on. Of course we had a minimal level of cross-training to avoid a single point of failure (due to sickness or project overloads), but specialization was the new normal, and it worked.   Perhaps it is time to look at market research job descriptions on a broader scale and consider how best to define positions to maximize effectiveness and employee satisfaction?

Please see our article on this topic, as published in Quirk’s:
http://www.quirks.com/articles/2013/20130125-2.aspx

[Do you have staff that could use some market research training? Check out our self-paced classes; most are under an hour, and all can be viewed conveniently from any web browser.]

But they also frustrate the suppliers.

Bearing in mind that there are literally over a thousand market research suppliers worldwide, most seek to do good work.  When they under-deliver, it is not intentional.  And they’re not rejoicing in slide creation parties.

So where is the disconnect?

As @edward04 commented on the Greenbook post: It takes two to tango.

Part of the challenge we have in market research is that suppliers often walk a very fuzzy line between being an outsourcing partner (takes on a project, completes start-to-end, full authority and responsibility) and being a staff extension (reports to the client, gets approval at major milestones, numerous points of shared responsibility).  MR agencies often are hired, and want to act, as an outsourcing provider—but in reality, it is a mixed role (especially compared to other outsourcing models). Not clear why this is an issue? Consider this example from the world of employee management:

I ran a market research agency for 13 years, and managed lots of researchers in that time.  I learned early on that employees cannot be successful if they are given responsibility, but no authority. It just doesn’t work.  Employees who can’t make decisions or solve problems related to their areas of responsibility can never be effective. The level of responsibility assigned to an employee must be matched by an equal level of authority to act.

In the case of market research engagements, I see a lot of cases where the client and the supplier have confusing or unspecified divisions of responsibility and authority.  Frustration for both sides is the result.

Is the Market Research Client-Supplier Relationship Unusual?

Clearly, there are always tensions between buyers and sellers—of any kind.  Buyers want more value, sellers want more margin. Buyers want it faster/more convenient, sellers want scalable processes. The list could go on and on.  And such conflicts are not all bad: a little constructive tension serves the purpose of making sure clients and suppliers help each other evolve.

As for articles such as the recent Greenbook post? Sure, it is fine to vent—for both clients and market research suppliers. Nothing wrong with that!  But the real issue is this: how do we step away from recycled platitudes about client-supplier relationships and make real change?  When do market research agencies and clients work best together? Perhaps when both parties precisely specify and agree to the relationship model: outsourced or staff extension.

 [The book, How to Hire & Manage Market Research Agencies, is available on Amazon in both paperback and Kindle editions]

Market research training classes: now with dates through March 2013.