Market Research

Rank Order vs. Point Allocation – Which Is The Better Option?

All survey designers, at some point in their career, will face the challenge of having to maximize information content for their clients while balancing the cognitive load we are asking of our respondents. This post examines two options for eliciting a respondent’s preference – rank-order versus point allocation exercises.

Rank order questions are in common usage across marketing and other forms of consumer-based research. The primary advantages offered by rank-order are multi-fold – they are easy to program and are available in most if not all survey platforms. This question format creates only a moderate amount of cognitive stress, unless the researcher goes beyond 5 – 7 items. Rank order questions are objective in nature and remove potential subjective bias.

This question format is not without some issues. By design, rank order questions produce ordinal data. This data type can present analytical challenges. For example, items can have the same median ranking. It is also possible that no item receives a top ranking, based on the median rank. You can assign the “most preferred flag” to the item with the greatest response percent at either the top or bottom position, depending on how your question is phrased to work around this.

The point allocation scale is a viable alternative to the rank order question type. In this scenario, respondents are asked to allocate 100 points across the slate of items for consideration. This question type is also known as a constant sum. The data produced by a constant sum question is interval-level, which means a full-slate of statistics can be applied against it. What makes it interval is the fact that an item can receive no points or all of the points. It is suitable for data procedures that require numeric data, e.g. regression or analysis of variance. Like the ranking problem, this question type is also objective in nature. It carries little risk of subjectivity bias. The image below shows what a respondent might see for a point allocation or a constant sum scale.

To convert this to a rank-order scale then substitute the following language:

Widget Image

Both methods fall into the domain of commonly accepted practice for survey research. Best practice for both variants is to limit the list to 5 – 7 items and to employ randomization (this minimizes first-order bias). Ultimately, the question format chosen should be a function of your client’s needs and your interest in analyzing the data.

The author: Greg Timpany directs the research efforts for Global Knowledge in Cary, North Carolina, and is partner in the firm Anova Market Research. You can follow him on Twitter @DataDudeGreg.

Greg Timpany is also a Research Rockstar Instructor.


Greg Alan Timpany

For more than 20 years Greg Timpany has delved into the world of marketing, analytics and strategy. His expertise lies in the space between the structured world of IT and the creative, customer-focused needs of marketing. His thought leadership has been honed by practical application in B2B, B2C and the public sector.

Greg’s experience has been gained by working in both client-facing agency roles and positions within internal marketing departments for companies such as The Los Angeles Times, Guitar Center, Wilkin Guge Marketing, and currently with Global Knowledge as their in-house subject matter expert on marketing research. In addition he is an adjunct professor of marketing at Meredith College, a blog contributor for Cvent, and co-founder of Anova Market Research.

Related Articles

Leave a Reply

Your email address will not be published.