Should a Customer Survey Ask People to Rank, Rate or Choose One Priority?

Choose one when the first priority is enough

A single-choice priority question fits a narrow decision: which issue should we discuss first, or which of these improvements would help most with a particular task? Name the task and the basis for choosing. ‘Which improvement would most help you send your next client proposal?’ is more specific than ‘Which feature is best?’

The tradeoff is deliberate information loss. You learn the selected first priority, but not the order or importance of everything left unselected. An item with few first-choice selections might still matter to almost everyone as a second priority. It might also matter intensely to a small group. Neither possibility can be resolved from first-choice counts alone.

Give people a truthful way out when none of the listed improvements would help, an important need is missing, or they lack enough experience to judge. Keep those meanings separate. ‘None would help’ challenges the candidate list; ‘I cannot judge yet’ calls for checking experience or understanding. Combining them into ‘Other’ makes the next investigation harder.

Rank only when the order below first place changes a decision

Ranking is useful when you need an ordering among a small, understandable set of alternatives and will use more than the first position. For example, a researcher comparing possible service packages might need to know which option people prefer if their first choice is unavailable. Ask for that order only when respondents can meaningfully compare the alternatives.

Treat full ranking as a demanding choice. The Australian Bureau of Statistics advises avoiding ranking questions because the task can be difficult, ties may not fit the format, and partial answers can be hard to interpret. It recommends individually labeled ratings when equal judgments and non-applicability matter.

If an ordered preference is still necessary, make the direction explicit and test whether people can distinguish adjacent items. A ranking of three genuinely comparable options can be more useful than an imposed order across ten unfamiliar ideas. If two items are equal in the respondent's view, forcing a tie-break adds a distinction you should not mistake for conviction.

Ranking tells you order, not distance. First and second may be almost interchangeable or far apart. Last place does not mean unwanted. And ranking only a person's top few choices does not reveal the order of the omitted items. Keep those limits in the analysis rather than assigning precise importance to positions.

Rate separately when equal or low importance is meaningful

Separate ratings fit questions about the degree of a problem, usefulness or satisfaction with each item. They let someone give two items the same judgment, or say every proposed improvement would have little value. A forced order cannot preserve those answers.

Choose a scale that matches the thing you need to learn. For an existing service, asking how much each problem interferes with a named task is different from asking how exciting a proposed feature sounds. Label the response options so the judgment has a clear meaning. Keep ‘not experienced’ separate from ‘experienced, but not a problem.’

Ratings may leave several items tied at the top. That is useful information if the needs are equally serious. It is insufficient if the business must choose one project next. In that case, investigate the practical constraint behind the tie or ask a focused first-priority follow-up. Do not quietly change the scale or remove the tie just to manufacture a winner.

The business still has to consider delivery cost, reach and dependencies. A customer preference is evidence about a need or choice; it is not by itself a release plan. A small change that resolves a serious obstacle may deserve attention even when a more attractive idea receives more votes.

Example: choosing the next improvement to a proposal service

Consider a hypothetical business that helps freelancers prepare client proposals. It is deciding among a clearer pricing table, reusable scope descriptions, a checklist of information to collect before writing and more visual themes. The immediate decision is which problem to investigate before funding development, rather than which feature to promise.

First, describe each candidate at a comparable level. ‘A clearer pricing table’ should say what would change, such as grouping optional fees separately from the base price. Give the other candidates equally concrete descriptions. Otherwise the question partly compares how persuasive the descriptions are.

Ask people who have recently prepared a proposal: ‘Which of these improvements would most help you prepare your next client proposal?’ Offer the four candidates plus separate answers for none helping, another improvement being more useful, and being unable to judge. Follow with an optional question: ‘What happened with your most recent proposal that explains your answer?’

This design identifies a starting point for investigation and the experience behind it. If pricing leads the responses but the explanations concern clients misunderstanding optional fees, investigate that confusion before redesigning the whole pricing page. If theme requests actually concern clients being unable to read the proposal on a phone, the named feature and the underlying problem are different.

A full ranking would be justified only if the team also needed the lower preferences. Separate ratings would fit a different question: how much each existing problem interfered with preparing the recent proposal. Those answers could show several serious obstacles at once. Choose the question to match the evidence needed, rather than collecting all three formats by default.

Read the response pattern before declaring a winner

For a first-priority question, report counts with their base: how many people answered, who was asked, and how many selected none, another need or inability to judge. Do not describe the most selected item as a majority unless it received more than half of the relevant responses. People who did not answer are not votes for the winner.

For ranking, inspect the positions as well as any summary. In an illustrative four-person result, item A might receive ranks 1, 1, 4 and 4, while item B receives 2, 2, 3 and 3. Both average 2.5. A splits first and last preferences; B sits in the middle. The same average hides different decisions about whom an improvement would serve.

Also check what a reported ranking score actually calculates. Jisc's Online Surveys documentation, for example, describes assigning points by position and averaging them. That is a summary rule, not a measurement of how much more someone values one item. Read the calculation before treating a score gap as an importance gap.

For ratings, examine how many people chose each response and how many had relevant experience. A high rating among a few experienced users and a moderate rating across many users pose different questions about reach and severity. Compare meaningful groups only when their experience is relevant and the group counts are visible; avoid broad customer claims from a small voluntary response set.

Diagnose an unhelpful answer before changing the format

If people choose ‘another improvement,’ examine the missing needs before adding more ranking positions. If they cannot judge, check whether they have experienced the task or understand the candidates. If everything is rated highly, ask whether the items are genuinely important together or whether vague wording made every improvement sound desirable. These failures need different repairs.

Test the actual question with people who resemble the intended respondents. Ask them to explain the task in their own words, describe what each candidate means, and tell you how they made their choice. Someone who ranks ‘reusable scope descriptions’ first because they expect a lawyer to review every proposal has answered a different proposition from the one the business intended.

Keep the candidate set and question wording with the results. Adding a strong new alternative can change the first-choice share of an unchanged item. Switching from ratings to a forced first choice also changes what is being measured. Neither shift establishes that customer needs improved or deteriorated.

Build a focused priority question in QuizFlow Labs

For a one-priority survey, use a Multiple Choice block in QuizFlow Labs and set Selection mode to Single select. This records one choice; allowing several selections answers a different question. Put the task and choice criterion in the prompt, then use distinct labels for the candidates and any valid non-choice answers.

Add an optional Text Response to collect the experience behind the choice. Keep the explanation focused on the decision you will make, such as the proposal problem to investigate. Read it alongside the selected option before deciding what to test. A useful decision flow connects the response to an appropriate next action instead of stopping at a popularity chart.

Start with one decision, one suitable answer format and one plan for interpreting it. If lower preferences would not change the next step, leave them unasked. If equal judgments matter, preserve them. If the result raises a new question, investigate that question before turning a preference into a promise.