When Should a Customer Survey Ask an Open-Ended Question?

Use choices for known distinctions and text for missing explanations

A multiple-choice question gives everyone the same distinctions to consider. That helps when you need to know which service someone used, whether they completed a task or which of several defined steps caused difficulty. The list still needs to include reasonable answers; structured does not automatically mean accurate.

Open text is useful when the missing information is the person's account, rather than a category you already understand. “What happened when you tried to get the file?” can distinguish an unreadable format from a confusing download link. Both might otherwise be filed under “download problem,” even though the fixes differ.

Pew Research Center describes using open questions in a pilot to discover answers that can inform a later fixed list. For a creator testing a new offer, that is a useful sequence: learn the language and distinctions first, then test whether a structured question captures them. Do not replace varied comments with categories merely to make a chart easier to produce.

Give the written answer a manageable job

“Tell us about your experience” leaves the respondent to choose the topic, time period and level of detail. Narrow the prompt to the information the decision needs. For example, “When you last tried to find a template, what, if anything, made it difficult?” identifies an occasion and allows a truthful answer that nothing was difficult.

Ask for the problem before demanding a solution. A request to “Describe the new search system we should build” asks customers to design the business's response. An account of what they searched for and what they could not find gives the team evidence to assess several possible fixes.

Keep the task proportionate. If a short account is enough, say “A brief description is enough.” Do not ask for a full history to diagnose one step. Government Analysis Function guidance recommends clear, balanced questions and testing the effort they require; “if anything” is useful when a problem is possible rather than assumed.

Avoid listing possible failures inside an exploratory prompt. “Was search slow, confusing or inaccurate?” supplies the categories you were trying to discover. Offer formatting help when needed, but distinguish help answering from suggestions about what the answer should be.

Choose the order according to what you want to learn

A written question after a structured choice can explain that choice. Asking which stage was difficult, then asking what happened there, keeps the follow-up focused. It also means the answer is given in the context of your list.

If you need an unprompted account of what stood out, ask for that account before presenting your explanations. Pew notes that earlier questions can influence later answers. Neither order is universally best: one supports explanation of a defined answer, while the other protects discovery from the categories you supply.

Be explicit about the scope when only some people receive a follow-up. Comments from customers who reported difficulty can help diagnose that difficulty. They do not describe everyone who completed the task, and customers who did not attempt it may have a different problem worth investigating separately.

An example: feedback on a template library

Consider a hypothetical creator who sells a library of presentation templates. The next decision is whether to improve finding a suitable template or help customers use one after downloading it. A general satisfaction score would not distinguish those two jobs.

The survey first asks which stage, if any, was most difficult on the customer's most recent attempt: finding a suitable template, downloading it, opening or editing it, or no difficulty. It also allows the person to say they have not tried using the library or that the list does not describe their experience. The wording asks for the main difficulty, so one selection is appropriate even if several stages were troublesome.

The written follow-up asks customers who selected a difficult stage what happened there. Someone might explain that the search results contained the right subject but no suitable page format. Another might explain that a downloaded file required software they did not have. These are invented possibilities, not findings.

The first account suggests investigating how format is described and filtered. The second suggests checking compatibility information before download. Neither alone justifies rebuilding the whole library. The creator checks the relevant journey, looks for similar accounts and tests a focused change before choosing a larger investment.

The tradeoff is coverage: asking for the main difficulty keeps the survey focused but may miss secondary problems. If the decision requires a complete record of every failed step, this short survey is insufficient; a follow-up conversation or observation of the task would be more suitable.

Plan how to read the answers before collecting them

Assign someone to review the comments and decide what evidence they will preserve. For a small survey, a simple review record can hold the original answer, the stage it concerns, a descriptive theme and a separate possible action. Keep what the customer said separate from what the team thinks should change.

Use themes that distinguish problems you could act on differently. “Template issue” hides the difference between finding a format and opening a file. Start with a manageable set of labels, read a varied batch, revise ambiguous labels and apply the revised definitions to earlier answers too. Keep unclear answers visible rather than forcing them into a confident category.

If several people review comments, have them label the same small batch separately and compare disagreements. A disagreement can reveal that “compatibility” includes both unavailable software and a genuinely broken file. Clarify that distinction before counting. GOV.UK's research analysis guidance similarly separates observations from interpretation and groups related evidence into themes.

Decide what a count means. If one answer describes the same failure three times, count that response once for that theme when reporting how many responses mentioned it. One answer may mention several themes, so theme totals can exceed the number of written responses. Unless you can establish unique respondents appropriately, describe counts as responses rather than distinct people. Preserve those rules with the summary.

Interpret silence, repetition and strong language carefully

A blank comment does not mean satisfaction, agreement or absence of difficulty. It means the survey has no written explanation from that person. Pew's research on its American Trends Panel found that willingness to answer open questions varied by question and respondent characteristics. Those findings are a reason to examine missing answers, not a benchmark for a small business survey.

Keep three quantities separate: submissions that reached the question, submissions with a substantive written answer and responses mentioning a theme. A theme’s share among commenters describes those comments. It does not establish the share of all customers experiencing the problem, especially when participation is voluntary or the question reaches only one path.

Repeated reports can justify checking a suspected failure, but repetition alone does not establish its cause. A single detailed account can also reveal a reproducible broken step worth fixing. Conversely, an angry paragraph may contain little diagnostic information. Assess what happened, what can be verified and what consequence followed, rather than ranking priorities by length or intensity.

When the remaining decision is “How widespread is this?”, use what the comments taught you to design a clear follow-up question for the relevant audience. That requires a suitable sampling and participation plan too. Converting themes into answer choices cannot, by itself, make a convenience survey representative.

Build the question pair in QuizFlow Labs

In a QuizFlow Labs decision flow, use Multiple Choice for a defined stage or category and Text Response for the person's explanation. Set single select when asking for one main difficulty; use multiple select only when you intend to collect several and can interpret that difference.

For an optional feedback comment, turn off Required response on the Text Response and test that leaving it empty permits progress. Describe it as optional in the prompt. Requiring text establishes that something was entered; it does not establish that the entry is a useful explanation.

Keep the prompt limited to the experience you need to understand. For a template-usage survey, there is no need to request passwords, payment details or identifiable information about a customer's clients. Explain how feedback will be used, and provide a separate support route if someone needs a personal reply. Do not imply that writing a survey comment opens a support case.

Test whether the answers will change a decision

Before sharing, ask someone representative of the audience to answer on the device they would normally use. Ask what they thought the prompt wanted and which experience they recalled. Check whether they can answer briefly without guessing what you want to hear. Pew's analysis of open questions found greater nonresponse for more demanding prompts; test the effort rather than assuming a text box is easy.

Then rehearse the analysis with a few clearly marked sample answers, including a blank, a vague complaint, two different problems and an account that fits none of the expected categories. If every answer leads to “improve the experience,” the prompt or review labels need more work. If different accounts lead to different investigations, the question is doing a useful job.

If answers mostly say “confusing,” check whether the prompt identifies a task clearly enough for someone to describe an event. If many accounts fit none of the categories, revisit the list rather than hiding them under “other.” If many comments are blank, test understanding and effort before adding more instructions or making the field mandatory. Different symptoms call for different changes.

Start with one explanation you genuinely need. Choose a focused prompt, give someone responsibility for reading it and state what would justify investigating, testing or measuring further. Open text earns its place when the business can turn a person's account into a better next decision.