I’m a fan of Google Consumer Surveys’ limitation on question length. Google limits you to questions of no more than 125 characters long, primarily – I believe – for a better experience for readers of the sites of its publisher partners (see In Search of an Answer).  The Google UI does point out that “Longer questions and answers reduce the quality of responses” and advises “Keep questions short and simple.”

Sadly, I recently encountered a Google Consumers Survey question that proved even short questions can go badly awry.

Respondents were asked “Will the $95 IRS penalty motivate you to shop this October for an Obamacare health plan?” and could choose “Yes,” “No,” and “Not Certain.”

Every day I quickly review about a dozen recently published surveys to choose a few to write about on Researchscape.com, setting aside ones like this with obvious errors. Good questions are hard to write, and I’ve written my share of bad ones. This question has quite a few problems:

1. The “$95 IRS penalty” is incomplete – The penalty is $95 per individual or 1% of household income, whichever is greater. By understating the penalty, this most likely depressed “Yes” votes.
2. The “in October” is overly specific – Some may purchase insurance in advance of October to avoid the penalty. This probably depressed “Yes” votes.
3. Obamacare is a loaded term – Originally coined as a pejorative by Republicans opposed to the Affordable Care Act, it has only gradually come to be used by Democrats. I’ve used Affordable Care Act in healthcare surveys I’ve conducted for precisely this reason. I’m not sure of the effect this would have on the overall results; it might increase nonresponse, for instance.
4. “Obamacare health plan” is too narrow – The intent was to find out if people would buy insurance because of the penalty, but they do not have to buy insurance from ACA health exchanges, as this wording may have implied. This also most likely depressed the “Yes” votes.
5. The sample is wrong – The question is asked of everyone, when it really should be asked of those currently without health insurance with a different question with new wording for those who currently have insurance. The user should have added a screening question with Google Consumer Surveys’ “Pick audience” functionality.

Whew, that’s a lot of mistakes for 16 words and 88 characters!

Questionnaire writing can be incredibly demanding, even when that questionnaire is only 16 words long.

Speaking of healthcare, maybe more Do It Yourself researchers should have their questionnaire’s vitals checked out by SurveyMedic?!

[This Guest Post was written by Jeffrey Henning of Researchscape. Be sure to follow him on Twitter at @JHenning]

Anyone dealing with digital images learns early on about the trade-offs of resolution — both high and low. While technology continues to advance, there is a constant here: Higher resolution gives more information — a clearer picture — but it also takes more space to store, and it’s slower to transfer electronically.

Well, curiously enough, there are similar trade-offs when it comes to market research survey design, and, in our case, “resolution” is defined by the number of points we choose for our scaled questions.

More Points Equals Higher Survey Research Resolution

How many points in a scaled question is “enough”? That’s a pretty common question in survey design. If you’ve spent some time with online market research surveys, you’ve probably noticed that they often have five points.  “Please rate your satisfaction on a scale from 1 to 5, with one representing ‘not at all satisfied’ and five representing ‘extremely satisfied’.” Sometimes, though, you may see a 7-point scale instead of 5. Is 7 points an improvement? How about 11? Is higher resolution better? The answer is, “that depends”.

Seeking Balance in Survey Scales

Ask this question of two researchers and you’ll likely get two very different opinions, but ideally we want to walk the line between simplicity for our survey takers and accuracy in our data. If we expect to see a lot of variability in the responses, higher resolution will allow our respondents to be more precise, providing better data. As an example, we tend to use larger scales for surveys about customer satisfaction because of the wide variation in customer experiences with a given brand.

Even or Odd Scales?

The choice of either an even or odd point scale really boils down to one issue: Do you want to give your respondents a “neutral” option? Market researchers tend toward odd scales for most projects, so 5 and 7-point scales are common, and even 11-point scales are seen occasionally.

There’s a debate, and it goes like this:

• Even forces a choice.  There’s no riding the fence with even scales — your respondent has to come down at least somewhat on one side of your question or the other. My personal take is that neutral can be a valid opinion and I don’t want to force people to say something that’s not true. And in some cases, people are too nice, so they might answer on the positive side — even if they don’t feel that way — unless you give them the option of neutral. It’s something of a quandary, and it requires some judgment on the researcher’s part.  Familiarity with the target population in question helps.

• Odd allows an easy-out.  Well, maybe. Some researchers believe that people will default to neutral because it’s easier, but there’s a debate about whether that really happens. In my experience, it is not an issue.  It used to be a problem years ago when many surveys were done on paper—we would call it “pencil whipping” (respondents would “whip” their pencils down the middle column of a survey grid).  But for online surveys in particular, I see no evidence that this is a notable issue.

Plan for Future Surveys

If you intend to be doing these surveys for a while, it’s worth thinking about picking a scale and sticking with it.  Even or odd, 4 or 7 points, standardize your system so that you have some consistency across your market research surveys over time. Some day you’re going to want to compare results, and having a standard format will make your life a little easier.

When in Doubt, Go Big

If you still can’t decide on a scale to stick with, err on the side of higher resolution. Just like in photographs, you can always collapse the scale when you’re doing your data analysis, but you can never expand it (increase the resolution) once you’ve captured the data.

Know Your Survey Goals

The choices really aren’t that complex. Larger scales allow a greater degree of potential accuracy, but require a bit more thinking by participants. As a researcher, you need to make these trade-offs carefully, so that your research delivers the best picture possible.

[Want to learn how to use questionnaires to capture attitudes and behaviors while avoiding the common mistake of over-relying on 5-point scaled questions? Check out our live, instructor-led class, Ask It Right: Choosing Scales & Answer Options for Online Surveys.]

Is Market Research a high-stress job?

It certainly can be.

But I can also tell you, after 25 years of doing market research, that there are known strategies for managing project-related stress.  My favorite is one Research Rockstar students hear me talk about frequently: the preemptive strike.

Most stress-inducers are due to unmanaged risk factors. Things that can go wrong in a market research project that cause budgets to be squeezed, timelines to slip or success criteria to be unmet.  Trying to manage these risks after they have already emerged is often difficult and highly stress-inducing.

Example of Market Research Project Stress

Consider the market researcher who has committed to a project where the screening criteria are known to be hard. Sure enough, at some point, it gets stressful.  Quotas aren’t being met, deadlines are looming, and the client (internal or external) is becoming unhappy. It is a brutal time, and stress levels soar.

In this scenario and others, the best option would have been a preemptive strike. That is, taking steps to make sure the problem never got to the stress-inducing point. In this example, there are 4 ways to do this:

1. Getting clients to pre-approve fallback strategies (what screening items could possibly be relaxed in the event it became necessary).
2. Having additional sample sources (and budget) on stand-by if needed.
3. Educating the client on the possibility that some quotas may not be met and why.
4. Having secondary sources that prove why some of the screeners are likely to be difficult (so that you can demonstrate that the possible data collection issue would not be due to your ineffectiveness, but to size-of-universe realities).

Market Research & the Preemptive Strike

Luckily, most market research risk factors that cause stress are known, and best practices exist to manage them. In many cases, the best strategy is to employ a preemptive strike as a way to neutralize risks before they blow up your project.

[Want to learn other ways to minimize risks and maximize efficiency in your market research projects? Click here to check out our Market Research Project Management 4-week Power Program!!]

By coincidence, I read two articles this past week on the theme of, “less is more.” These articles were not specific to market research, yet they do apply.

Less is More, for Market Research Credibility

In the February 2012 issue of Inc. Magazine, Twitter and Blogger co-founder Evan Williams promotes the idea of doing less. That is,  “If you have too many things to think about, you’ll get to the superficial solution—not the brilliant one.”  For we researchers, this is a hard balance. On one hand, we know that the value of analysis seldom comes from focusing on one or two data points—it comes from identifying recurring themes and patterns. Indeed, we often talk about “weaving” together a story from multiple data points.  But we also know that at the other extreme, dumping too much data in a client’s lap, leads to disaster: they turn off, stop listening, and even judge us as unable to prioritize or synthesize, thus hurting our profession’s credibility.  Allowing people to focus more on fewer items, does enhance how market research is perceived.

Less is More, for Richer Market Research Analysis

In the Sunday New York Times (January 20^th, 2013 edition), Matthew E. May wrote about, “The Art of Adding Through Taking Away.”  The article points to the strength of this wisdom through ancient proverb and more recently by quoting Jim Collins, who apparently observed that, “A great piece of art is composed not just of what is in the final piece, but equally important, what is not. It is the discipline to discard what does not fit — to cut out what might have already cost days or even years of effort …and marks the ideal piece of work, be it a symphony, a novel, a painting…”

This is so true for those of us who write market research reports. It is always a challenge to hold back—we find so many interesting and tempting data points in a single study. Yet we know that the discipline to reduce our work to its core essence is essential, and will even help us to create more meaningful analyses. A good market researcher will find many interesting things to report; a great one will focus on fewer items but bridge the gap to actionable insights.  The restraint is not easy, but is always rewarded.

[Report writing strategies are covered in Research Rockstar’s Project Management class. Next session starts February 28^th and meets once a week for 4 weeks.]

A little reflection on the time and knowledge demands of present day market research suggests that it may be time to update how the function is structured and staffed. Given that there are now many different skills involved in market research, and that each of these skill areas is increasing in complexity, how can one market research project manager excel? How can a general market research project manager have adequate skills, time and talent? Maybe it’s time to let people do what they do best. Or alternatively, perhaps assign them by the way their work will be applied?

While running a market research agency years ago, I learned the importance of specialization the hard way.  Sometimes it was better to take somebody who was a market research project manager and have them just focus on statistics—if that’s really where their talent lay.  Over time, we actually restructured the organization this way—a Copernican revolution, if you will, that placed human talent at the center of consideration. We had a survey programming specialist, a data collection specialist, an editing specialist, a graphics specialist, and so on. Of course we had a minimal level of cross-training to avoid a single point of failure (due to sickness or project overloads), but specialization was the new normal, and it worked.   Perhaps it is time to look at market research job descriptions on a broader scale and consider how best to define positions to maximize effectiveness and employee satisfaction?

Please see our article on this topic, as published in Quirk’s:
http://www.quirks.com/articles/2013/20130125-2.aspx

[Do you have staff that could use some market research training? Check out our self-paced classes; most are under an hour, and all can be viewed conveniently from any web browser.]