Do these websites matter? Aren’t they just electronic billboards?

I think they do matter.  A lot of people who buy and use market research visit agency websites. And frankly, they are more likely to visit an agency site than that of a professional association (such as the MRA or CASRO) or of a trade publication (such as Quirk’s or Research Magazine). So these agency websites are, intentionally or not, a key source of client education.

And at the risk of sounding preachy, I think all market research professionals have a responsibility to promote market research ethics, professional standards, and innovation. And one of the ways to do it is through our public faces.

The Method to My Madness

I selected 6 websites from the Honomichl list and applied my 4 criteria. To refresh, the criteria are:

1. Educational content.
2. Standards and ethics.
3. Friendliness.
4. Innovate thinking.

Grades of A-D were assigned based on specific criteria within each metric. As an example, an agency that has at least 20 educational white papers, videos or similar forms of content got an A on the Education metric. As another example, an agency that has no mentions of market research ethics or adherence to ethical standards got an F on that metric. And if I had to search for an item for an hour only to find it buried in press releases, that gets a D. Please note that in some cases I may have missed relevant content—but if I spent an hour seeking out content for a given metric and could not find any, it is for all purposes, a low grade.

So how did the agencies do? I selected the following agencies from the Honomichl list:

Burke

IPSOS

Maritz

Market Probe

Synovate

TNS

And to be clear, I am not looking to pick on anyone. I did this as an exercise to see how well agencies do on these metrics.  My key take-always:

• Educational. Incredibly inconsistent, though a few shining stars.
• Standards and ethics. As a group, agencies do this poorly. None had mentions of ethics or standards on their home page or other prominent pages on their web sites.  Professional association badges are hard to find, if evident at all.
• Friendliness. Again, very inconsistent.
• Promotes innovative thinking. Very inconsistent. A couple of agencies do a good job; a couple seem to neglect this angle altogether.

So How Did They Do?

The top 2 grades go to….drum roll….IPSOS and TNS.  Both have web sites that have the highest scores on the metrics applied. IPSOS has a bit more content that promoted market research innovation. But both sites do a great job on education and friendliness.

I welcome any feedback here, or by phone: 508.691.6004 ext 705. Thanks!

But if you have ever looked at more than 2 or 3 agency web sites, you will see that the quality and content vary dramatically. Almost dismayingly so.

So how to compare them? How to decide if an agency is worth further consideration?

I recommend using the following 4 criteria to assess market research agency websites:

Educational content. Number of white papers, videos, blogs and other content that educates client-side visitors. An agency that invests in client education is an agency that values its client relationships

Standards and ethics. Visible statements about market research ethics, and adherence to standards promoted by the industry’s professional associations such as CASRO, the MRA and ESOMAR. Choose agencies that have solid credentials; alas, there are many that do not.

Friendliness. Based on a site’s overall readability and ease of navigation. Why hire an agency that can’t communicate about research in an accessible way?

Innovative thinking. Has staff-authored books or papers on innovative topics, promotes new methods on website or blog. Perhaps even shares agency-funded “research on research.” An agency that spends some time and money investing in research methods will be able to best advise you on what methods will best meet your needs; you don’t want to go to a doctor that hasn’t been trained in 20 years, do you?

Obviously, a market research agency’s website is only one way to become exposed to its capabilities. Still, it is an important one. To find links to market research agencies and their websites, try the AMA’s Green Book, Quirk’s directory, the MRB Directory, or for a more global listing, ESOMAR’s online directory.

[This is part 1 of a 3-part blog series on market research agency web-sites: the next post will cover some of my favorite agency websites which meet the criteria above. Be sure to get the RSS feed so you don't miss parts 2 and 3: subscribe.]

But sometimes the incremental time, cost and sweat of that extra effort simply doesn’t make sense. Sometimes, “good” is just perfect.

I was reminded of this last week at the Launch Camp conference in Cambridge. The event, for entrepreneurs seeking social media wisdom, had some interesting speakers; the one from whom I learned the most was Dharmesh Shah, Chief Technology Officer and Founder of HubSpot (on Twitter as @Dharmesh).

In three years, this company has gone from start-up to 2,000+ customers, most of whom pay a monthly fee. Dharmesh shared his start-up success insights at Launch Camp and advised the attending entrepreneurs to focus on practical marketing. Selling stuff. Tracking key metrics to understand what sells stuff. And in his case, this clearly works.

He observes that many entrepreneurs get bogged down by over-analyzing their decisions—ultimately missing their window of opportunity. Key areas for such analysis paralysis? Product optimization and pricing.

ACK! Product concept testing and pricing research are two key pillars of market research practices around the world! But of course, he is correct. Especially in the context of new or rapidly evolving product categories.

Product Concept Testing

Market research offers proven methods for testing new product concepts—methods that can prioritize features or optimize feature-price combinations. And that’s great.

But I have seen companies completely miss windows of opportunity because they kept adding on less-than-critical features before they would launch. Kept conducting more and more research to inform (or justify) their decisions. Their leaders traded early market feedback for an over-engineered product. Dharmesh chastised this approach and emphasized that while market research is useful, at some point you need actual market feedback in order to inform further improvements. The ultimate feedback: will people buy it? If they buy it, will they return it?

Of course, these days, there are ways to simulate actual product releases to do this—although that is not a realistic option for all categories.

Pricing Research

Look, if you are talking about mature consumer product categories (like toothpaste and laundry detergent), pricing research is a very defined, concrete sort of practice. But in many B2B markets, emerging markets, and new product categories, it simply isn’t perfect. Yes, do some research. Do some primary research, analyze competitive/substitute pricing, understand your target market’s overall budget, know your expected ROI. But at some point you have to take a leap with pricing. And as Dharmesh said, despite long-held tenets to the contrary, you CAN adjust your pricing down the road.

Imperfect Data is Better Than No Data

Yes, it is true—imperfect data is better than no data. And sometimes, directional data sooner is better than quantitative data later.  In any case, knowing when to stop conducting market research in order to price and release new products can be tricky. Luckily for busy professionals seeking to inform product and pricing decisions, there are many options along the continuums of research speed and exactitude.

BTW, Dharmesh has a book out—I ordered my copy and can’t wait to read it: Inbound Marketing.

[Would you rather take one market research class for $2000 or get unlimited access to 12 online for $600/year? Or how about 5 for FREE? I thought so!  Sign up for a Research Rockstar membership today: http://is.gd/87vvd]

[For more info on Launch Camp search #LaunchCamp on Twitter for great links to blogs, RTs and even videos from the event]

Using price as the selection basis is simple and comfortable.

So here’s the catch: if you are sending RFPs only to agencies that come back to you with similar ideas, it’s time to rethink your short-list.

These days, there are innovators out there. Agencies using cooler tools, applying newer sample quality processes, and even offering new deliverables. Their methods may push you out of your comfort zone. Their proposals may be harder to read because they won’t be full of the same boilerplate you’ve been seeing for years.

But if you want research that will have an impact, break you out of a market research rut, and create more excitement among your internal clients at the end of the day, it’s time. Create a new agency short-list that includes innovative firms, perhaps even a few that you are skeptical about. You may be pleasantly surprised at the options they offer you.

[If you’d like help rethinking your agency short-list, I can help. Contact me at 508.691.6004 ext 705, or KKorostoff@ResearchRockstar.com]

[Want more market research ideas? Subscribe to the newsletter: FREENEWSLETTER]

But when comments I share that are “pending moderator review” never appear, it really annoys me.

About 10 days ago, I read an interesting article on TMCnet—a site that I like for technology-related topics. But it just so happened that this article had some important omissions. So I posted a thoughtful reply. Nothing incendiary. Nothing rude. Just a friendly sharing of information with the author and fellow readers.

It never appeared.

After a week, I emailed the editor. Still nothing.

The original article recommends NPS (Net Promoter Score) as the optimal standard for customer satisfaction with telecommunications providers. Ummm, no. So since I didn’t get to share on the TMCnet site, let me share some information here for those of you interested in measuring customer satisfaction in the telecommunications space.

• “There are many scenarios in which customers may be satisfied with certain service levels or offerings yet refrain from recommending or referring the larger offering to their friends.” Yes, this is very true.
• “…customer referrals – should be the ultimate measure of customer satisfaction and should be cultivated to the greatest extent possible.” Not necessarily.

In telecommunications, willingness to refer is not always the best metric. Having done over a hundred research studies on telecomm topics over the past 20+ years, I know that other items can be more relevant. For example, two items that are very important in the telecomm space:

1. Willingness to renew (vs. propensity to brand switch). For some service providers, lack of brand loyalty is a huge challenge. And cost of customer acquisition can be quite high. So for them, the most useful metric can be renewal intent.
2. Interest in “add-ons” (incremental features/services that would increase $/customer). Again, because the cost of customer acquisition can be high in telecomm, some service providers focus not only on retention but on extensions; how can we sell more to the existing customer base? That’s why in telecomm you often hear people talk about raising ARPU (average revenue per user). And customers’ willingness to buy more says a lot (like how well the proposed add-ons align with their interests, and how far the brand has permission to extend).

Yes, NPS is a wonderfully efficient approach to measuring customer loyalty. But it isn’t the only one. Customer satisfaction and loyalty research is not a one-size-fits all proposition. Telecomm providers need to take the time to identify the best metrics for their research to be truly useful.

[As always, please add a comment or question here, or call the Blog Requests line (508.691.6004). Thanks!]

I have used this diagram to walk market research newbies through the process many times, and it always gets rave reviews. While the chart is pretty self-explanatory, a few items are worth pointing out:

1. Two major steps (questionnaire design and deliverables) are iterative.  I always point out that for both of these steps, you need to plan for at least 2 and usually 3 rounds leading to final, approved versions. Inevitably, this leads to a discussion about roles and responsibilities—which is great.
2. I always stress that during the data collection process, we need to take time to document an agreed upon analysis plan and deliverables plan. Since I am often using this with folks new to market research, this leads to an important discussion about options, which is great to have well before a project starts.  Occasionally I get push back on the concept of a “deliverables plan.” But given all the ways of delivering data these days, and the unique needs of different audiences, having a plan documented is great way to avoid last-minute panic.
3. And yes, I really do talk about celebrating. Maybe that sounds trite, but with folks new to research, adding a little fun to the process scores some points. And let’s be honest; at the end of a big quant project, a little celebration is in order!

Please feel free to use this chart. If you want me to email you a separate image, just send me a message (KKorostoff@ResearchRockstar.com).

[If you like the blog posts here, you will love the newsletter! Not yet subscribed to the Research Rockstar newsletter? Sign-up here: SIGNUP]

That’s right. Two.

There’s the stated research objective. Perhaps something likes, “Determine current levels of brand awareness in 5 key market areas,” or “Test 6 potential new marketing messages for alignment with emerging customer needs.”

Then there’s the other, assumed objective: getting engaged respondent participation. This is an implicit objective that too often gets minimized. Yes, we all know we have to do everything we can to maximize response rates, but the choice of methodology is too often driven by the research objective—not the respondents’ needs.

As researchers, we talk a lot about matching the methodology to the first objective. But given low response rates and the preciousness of qualified respondents, we need to focus a lot more on matching the methodology to the audience.

An Example

A researcher I know from a software company was upset after working with a market research agency on a huge study of IT executives. They collected over a thousand responses to an online survey, but data collection was brutally slow due to low response rates. When she finally got the data, she had a lot of important items to which there were a surprising percent of neutral or even “don’t know” responses. Putting aside that this issue should have been caught during the pre-test phase of the project, this was hugely disappointing.

I looked over the screening criteria myself, just to see what the scoop was, and it was obvious that the audience they were targeting was too senior for the 25-minute, very technical, online survey. The topic was about a fairly new technology, so chances are they were interested in the topic—but the methodology choice and level of detail was wrong.

The Shadow Objective

It’s always there. The need to match the project’s methodology with the target respondents’ preferences and behaviors. Maybe you want quantitative data, but the target group gets too many similar requests as is. Maybe you want to do focus groups, but your target population works in a field where scheduling is too uncertain for them to commit to 2 hours of time. Maybe you want to do a phone survey, but your audience has a low penetration of landlines.

Bottom line

Choosing the best methodology for any research study requires considering the project’s objective and the shadow objective. The good news? These days there are so many methods and tools that can make the research experience engaging, there is no need to be constricted by the choice of survey versus focus group.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

• When you ask questions, do they listen and respond respectfully and credibly?
• When you ask for recommendations, are the responses thoughtful?
• Do they come to you proactively with suggestions and ideas?
• When you read their reports and memos, do you feel the information is reliable and credible?

I recently spoke with a long-time client who confessed that the only reason he hasn’t fired his market research agency is fear of change. For him, it’s the “devil you know…” syndrome. But because he doesn’t trust his agency, he is spending an inordinate amount of time checking their work and documenting communications via memos.  I’m working on a new short-list for him so that at least he can see what his agency options are. Will he take the plunge? I’m not sure. But I feel compelled to encourage him to look.

There are hundreds of market research agencies and hundreds of market research consultants. Changing agencies can be painful, but you deserve an agency that you can trust with your valuable market research investments.

Need to rethink your agency choices? Interested in finding agencies that offer more methodology options? Check out the directories at the MRA (http://www.bluebook.org/index.cfm) and Quirk’s (http://www.quirks.com/directory/index.aspx). Want help creating a new short-list of potential providers? Contact me at KKorostoff@ResearchRockstar.com. Thanks!

If you allow employees to invite customers to take customer satisfaction feedback surveys, make sure they aren’t saying things like, “The highest score is a 10. I hope I earned a 10!” You have probably experienced this yourself, perhaps at an auto dealership or retail chain. Upon completion of your transaction, the clerk give you the feedback survey instructions (perhaps advising of a phone call you will receive, or giving you a URL to use), and then says something like, “I’m hoping you rate me as “extremely helpful.”

Obviously, such behavior not only taints the data, it can also make customers uncomfortable.

So how can you determine if your customer feedback system is being abused? Here are three easy options:

• Simple analysis. For example, if you are in retail you may be able to run the statistics on individual sales people or customer service reps to see if their scores appear artificially high or simply too consistent.
• Get outside help.  Deploy some mystery shoppers on a discovery mission. What do they experience?
• Ask. In the feedback survey itself, ask respondents if anyone told them how to respond or suggested a desirable feedback score.

[OH NO! The Research Rockstar RSS feed self-destructed in December. So if you have not re-subscribed recently, please click here for RSS or email updates: SUBSCRIBE]

If you do an annual employee survey to gather current perceptions and attitudes—great! This is an area of research that can uncover real opportunities to boost employee satisfaction and productivity, and even identify ways to save money. But do be sure to include technology-related content—especially if your employees use computers, communications and related technologies as part of their jobs.

Not sure where to start? Take an hour to brainstorm some tech-related hypotheses. Before you know it, you will have a good set of items worthy of investigation. Here are some examples that are relevant to many organizations today:

• Our employees dislike using their desk phones; they prefer to forward their calls to their mobile phones
• Our employees are not following back-up protocols consistently
• Employees need more training on new applications
• Employees want to use (insert application or web site) but the company currently blocks it
• Employees want more leniency about what they can choose to install on company-provided computers
• Many people find the current IT help desk processes confusing, so they tend to avoid it until a problem becomes critical
• Employees are running blogs that sometimes touch on business topics, but are unaware of what disclaimers and protocols they should follow

Of course, the point is not to find out what they want and instantly provide it. There are obvious reasons why, for example, certain websites are blocked. Still, keeping an open mind and hearing this feedback may suggest a need to better explain why the policy exists or perhaps even identify situations where exceptions should be made.

Just Because You Build It, Doesn’t Mean They Will Come

Companies spend a lot of money on technology in an attempt to boost employee productivity—but how many go back to make sure it has happened? I have seen cases where employee research has uncovered surprising results—like that employees aren’t even using a telephony feature assumed to be critical, or that they avoid a specific business process because the related application is too cumbersome.

Does adding technology-related content add too much?

If the new content would make your annual employee survey too lengthy or onerous, then consider your options. Perhaps a smaller percentage of employees can be asked to complete the additional questions. Or maybe the employee base can be divided and asked to take 2 different questionnaires. Perhaps the questionnaires can be separated and done 6 months apart—so that employees don’t feel bombarded. Or maybe the tech questions can be tackled through in-house focus groups, for a more qualitative and in-depth discussion.

In any case, as with all employee research, it’s critical to make sure that upon completion, employees are thanked and next steps identified. Communicating how the research is driving specific, concrete actions that will improve the workplace will go a long way to boosting employee satisfaction. In contrast, employees who share thoughtful feedback and then see no action will be less willing the next time you ask.

[Is this a topic of interest? Interested in some help adding IT-related content to your employee research program? Contact me at kkorostoff@ResearchRockstar.com]