I’m a fan of Google Consumer Surveys’ limitation on question length. Google limits you to questions of no more than 125 characters long, primarily – I believe – for a better experience for readers of the sites of its publisher partners (see In Search of an Answer).  The Google UI does point out that “Longer questions and answers reduce the quality of responses” and advises “Keep questions short and simple.”

Sadly, I recently encountered a Google Consumers Survey question that proved even short questions can go badly awry.

Respondents were asked “Will the $95 IRS penalty motivate you to shop this October for an Obamacare health plan?” and could choose “Yes,” “No,” and “Not Certain.”

Every day I quickly review about a dozen recently published surveys to choose a few to write about on Researchscape.com, setting aside ones like this with obvious errors. Good questions are hard to write, and I’ve written my share of bad ones. This question has quite a few problems:

1. The “$95 IRS penalty” is incomplete – The penalty is $95 per individual or 1% of household income, whichever is greater. By understating the penalty, this most likely depressed “Yes” votes.
2. The “in October” is overly specific – Some may purchase insurance in advance of October to avoid the penalty. This probably depressed “Yes” votes.
3. Obamacare is a loaded term – Originally coined as a pejorative by Republicans opposed to the Affordable Care Act, it has only gradually come to be used by Democrats. I’ve used Affordable Care Act in healthcare surveys I’ve conducted for precisely this reason. I’m not sure of the effect this would have on the overall results; it might increase nonresponse, for instance.
4. “Obamacare health plan” is too narrow – The intent was to find out if people would buy insurance because of the penalty, but they do not have to buy insurance from ACA health exchanges, as this wording may have implied. This also most likely depressed the “Yes” votes.
5. The sample is wrong – The question is asked of everyone, when it really should be asked of those currently without health insurance with a different question with new wording for those who currently have insurance. The user should have added a screening question with Google Consumer Surveys’ “Pick audience” functionality.

Whew, that’s a lot of mistakes for 16 words and 88 characters!

Questionnaire writing can be incredibly demanding, even when that questionnaire is only 16 words long.

Speaking of healthcare, maybe more Do It Yourself researchers should have their questionnaire’s vitals checked out by SurveyMedic?!

[This Guest Post was written by Jeffrey Henning of Researchscape. Be sure to follow him on Twitter at @JHenning]

Leave a Comment

It’s a bit of dogma often claimed by market researchers, though it is circulated in other fields as well (including software development).

Why is it so popular? Primarily because it does pass the gut test: market research projects that are fast and good are unlikely to be cheap. Ones that are cheap and fast are unlikely to be good, and so on.

However, the dogma has become outdated. Thanks to new technologies and methodologies, market research projects are being completed faster and less expensively than they once were—and are still meeting quality needs. Or to employ yet another over-used aphorism, we have learned to do more, with less.

Of course, we still make trade-off’s in designing and executing market research studies. Just different ones.

Today, the balancing act is no longer about cheap, fast or good. Today it is about projectable, innovative or simple.

We often want research that is projectable, such that it accurately describes a target market or population of interest.

We also often want research that is simple, such that its risks can be managed and it can be delivered according to intended budget and schedule parameters.

And these days, we are also often seeking innovation. Not for the sake of innovation itself, but for the sake of overcoming known issues such as the limitations of self-reported behavior and the need for more accurate emotional measures.

Yet these are trade-offs.  Cool new methods and tools that help us gain rich projectable insights into emotions aren’t simple.  Market research projects that are simple and innovative are rarely projectable. And based on my experience, those that are simple and projectable are rarely innovative.

What do you think: Is the current trade-off between projectable, innovative or simple? And if so, can you think of a project you have done that was, in fact, all three? Is it possible?

[Get the best market research articles of 2012 in our convenient compendium. Topics include Facebook for research, Google consumer surveys, mobile research and more.]

Leave a Comment

In catching up on market research reading, we stumbled on this little gem from Question Science Blogspot.  In this article, Jon Puleston tells us about some surprising statistics he overheard while attending the ESOMAR 3D conference at the end of 2012:

350 out of 36,000—Porsche culled through 36,000 social media responses and found that only 350 were “useful”. Significantly, all of the comments were processed manually. This suggests that deciphering data from social media could be a poor investment.  So, can text analytics software accurately decipher social media comments, and are the comments even worth deciphering?   Clearly, this is going to vary by topic, brand in question and scope. Some brands/keywords get a lot more “garbage” than others.  What we have found here at Research Rockstar is that you have to do some serious testing of your topic/brand name/keywords of interest before you invest significantly in social media analysis.

240 hours– The amount of time spent by a market research firm analyzing text from 1000 Facebook users.

.18—A survey by Jannie Hofmyer and Alice Louw from market research company TNS, showed a surprising lack of correlation between “aided awareness of a brand & purchase activity”. Their research revealed that surveys are routinely constructed incorrectly and contain questions that are incapable of measuring behavior. Customers and non-customers of products should take different surveys to create relevant survey data results.

50%– Peit Hein van Dam, from digital tracking company Wakoopa, tracked a 50% variation between the claimed readership level of a Dutch newspaper and the readership level tracked on mobile devices and computers. “Cookie” tracking proves to be largely inaccurate in counting unique visitors and web traffic.

[Want access to more market research articles and training materials?

Sign up for the Research Rockstar newsletter: SIGNUP]

Leave a Comment

Do you prefer American Marketing Association (AMA) events or Marketing Research Association (MRA) events?

This is a questions I get frequently from Research Rockstar students and other people I run into—even just recently at the MRA’s Joint Chapter conference in Miami.

Who has time or money to attend two or more major events a year—if that? Other than companies with booths and selling things at the events, not many.

Both the AMA and MRA offer great annual events, and additional conferences throughout the year.

The comparison I most often get asked about is AMA versus MRA annual conferences. I will share my experiences with these events, and hopefully this will be helpful.

At the annual AMA market research conference, I meet a mix of market research and marketing professionals. Personally, I like that mix—it leads to interesting discussions and prevents us from treating market research as a silo. The marketing people tend to be those who have roles that mix market research and broader marketing responsibilities.  The market research attendees are a good mix of client-side and supplier-side researchers.  The presentations at this conference tend to showcase a lot of case studies, and cover a mix of market research standard methods and emerging solutions.

Bottom line: for great networking with a diverse crowd and broad learning, the AMA conference is an excellent choice.

At the national/annual MRA conference, I meet a variety of market research professionals; client-side researchers, suppliers, and even some educators.  In the past, these events felt a bit too skewed to the supplier side, but that has changed a lot over the past three years.  The event now gets a nice representation of client-side researchers—people who can talk firsthand about the experience of buying, managing and delivering in-house research.  To me, this is important: without a mix of attendee types (clients and suppliers), the networking just gets too dull.  While different than the annual MRA conference, even at last week’s joint chapter conference in Miami, I saw great presentations and met several people from client-side market search departments.

The annual MRA event is also great for a mix of case study presentations (showing actual market research applications) and in-depth learning related to new methods.

Bottom line: for great networking and in-depth learning, the MRA conference is an excellent choice.

[Update: it appears that the AMA may be discontinuing its annual market research conference? I will update as I learn more!]

[Want to receive market research news and articles by email? Click here to sign up for Research Rockstar's newsletter.]

Leave a Comment

Anyone dealing with digital images learns early on about the trade-offs of resolution — both high and low. While technology continues to advance, there is a constant here: Higher resolution gives more information — a clearer picture — but it also takes more space to store, and it’s slower to transfer electronically.

Well, curiously enough, there are similar trade-offs when it comes to market research survey design, and, in our case, “resolution” is defined by the number of points we choose for our scaled questions.

More Points Equals Higher Survey Research Resolution

How many points in a scaled question is “enough”? That’s a pretty common question in survey design. If you’ve spent some time with online market research surveys, you’ve probably noticed that they often have five points.  “Please rate your satisfaction on a scale from 1 to 5, with one representing ‘not at all satisfied’ and five representing ‘extremely satisfied’.” Sometimes, though, you may see a 7-point scale instead of 5. Is 7 points an improvement? How about 11? Is higher resolution better? The answer is, “that depends”.

Seeking Balance in Survey Scales

Ask this question of two researchers and you’ll likely get two very different opinions, but ideally we want to walk the line between simplicity for our survey takers and accuracy in our data. If we expect to see a lot of variability in the responses, higher resolution will allow our respondents to be more precise, providing better data. As an example, we tend to use larger scales for surveys about customer satisfaction because of the wide variation in customer experiences with a given brand.

Even or Odd Scales?

The choice of either an even or odd point scale really boils down to one issue: Do you want to give your respondents a “neutral” option? Market researchers tend toward odd scales for most projects, so 5 and 7-point scales are common, and even 11-point scales are seen occasionally.

There’s a debate, and it goes like this:

• Even forces a choice.  There’s no riding the fence with even scales — your respondent has to come down at least somewhat on one side of your question or the other. My personal take is that neutral can be a valid opinion and I don’t want to force people to say something that’s not true. And in some cases, people are too nice, so they might answer on the positive side — even if they don’t feel that way — unless you give them the option of neutral. It’s something of a quandary, and it requires some judgment on the researcher’s part.  Familiarity with the target population in question helps.

• Odd allows an easy-out.  Well, maybe. Some researchers believe that people will default to neutral because it’s easier, but there’s a debate about whether that really happens. In my experience, it is not an issue.  It used to be a problem years ago when many surveys were done on paper—we would call it “pencil whipping” (respondents would “whip” their pencils down the middle column of a survey grid).  But for online surveys in particular, I see no evidence that this is a notable issue.

Plan for Future Surveys

If you intend to be doing these surveys for a while, it’s worth thinking about picking a scale and sticking with it.  Even or odd, 4 or 7 points, standardize your system so that you have some consistency across your market research surveys over time. Some day you’re going to want to compare results, and having a standard format will make your life a little easier.

When in Doubt, Go Big

If you still can’t decide on a scale to stick with, err on the side of higher resolution. Just like in photographs, you can always collapse the scale when you’re doing your data analysis, but you can never expand it (increase the resolution) once you’ve captured the data.

Know Your Survey Goals

The choices really aren’t that complex. Larger scales allow a greater degree of potential accuracy, but require a bit more thinking by participants. As a researcher, you need to make these trade-offs carefully, so that your research delivers the best picture possible.

[Want to learn how to use questionnaires to capture attitudes and behaviors while avoiding the common mistake of over-relying on 5-point scaled questions? Check out our live, instructor-led class, Ask It Right: Choosing Scales & Answer Options for Online Surveys.]

Leave a Comment