Skip to content

Biased and bad survey questions (and how to fix them)

Ten kinds of biased survey questions with a fixed version of each, eight examples of confusing survey questions, the answer-option mistakes that skew results, and a checklist to run before you send.

Ethan Brooks, Software Engineer

By , Software Engineer at Florm

Published · Editorial policy

Types of biased survey questions with examples and fixed versions
TypeProblemBiased exampleBetter version
Leading questionHints at the answer the author wants.How much did you love our new checkout?How easy or difficult was the new checkout to use? (1 Very difficult – 5 Very easy)
Loaded questionContains an assumption the respondent may not share.What do you dislike most about your commute?How would you describe your commute? (Rating scale), then an optional “What would improve it?”
Double-barreled questionAsks about two things but allows one answer.Was the staff friendly and helpful?Ask “How friendly was the staff?” and “How helpful was the staff?” separately.
Absolute questionUses always, never, all or every, which few people can say yes to.Do you always check the price before you buy?How often do you check the price before you buy? (Never / Rarely / Sometimes / Often / Always)
Vague questionUses words that mean different things to different people.Do you exercise regularly?On how many days in the past 7 days did you exercise for 30 minutes or more? (0–7)
Jargon or acronymsAssumes knowledge the respondent may not have.How satisfied are you with our SLA on P1 tickets?When you reported an urgent problem, how satisfied were you with how fast we responded?
Double negativeMakes people work out what “yes” means.Do you disagree that the office should not be closed on Fridays?Should the office be open on Fridays? (Yes / No)
Agree/disagree statements onlyPeople tend to agree with statements regardless of content (acquiescence bias).“The training was useful.” Agree / DisagreeHow useful was the training? (Not at all useful – Extremely useful)
Socially desirable questionMakes one answer look better, so people give it.Do you recycle, like most responsible people?Which of these do you usually do with empty bottles? (Single choice, including “Put them in general waste”)
Assumptive questionAssumes a behavior that may not apply.How often do you use our mobile app?First ask “Have you used our mobile app?” (Yes / No), then ask how often only if yes.
Examples of confusing survey questions and clearer versions
ProblemConfusing questionClearer version
Two time framesHow satisfied were you last year and this year?Ask about one period: “How satisfied are you today?”
Unclear unitHow much do you spend on lunch?“On a typical workday, how much do you spend on lunch, in dollars?”
Hidden conditionIf you used support, would you use it again?Ask “Did you contact support in the past month?” first, then the follow-up.
HypotheticalWould you pay more for faster delivery?“Which delivery option did you choose for your last order?” plus a price-choice question
Long and nestedThinking about the last time you visited, and considering both the website and the app, how would you rate it?“How would you rate your last visit to our website?” and a separate question for the app
Memory overloadHow many emails did you get from us this year?“About how often do you get emails from us?” (Never / Monthly / Weekly / More often)
Unclear “you”How do you rate the team’s performance?Name the team: “How would you rate the support team’s response time?”
Overlapping optionsAge: 18–25, 25–35, 35–45Age: 18–24, 25–34, 35–44, 45 or older, Prefer not to say
Answer option problems that bias survey results
ProblemWhat to do
Unbalanced scaleExcellent / Very good / Good / Fair offers three positive answers and one neutral one. Use a mirrored scale: Very poor / Poor / Fair / Good / Very good.
Missing optionsIf someone’s answer isn’t listed, they pick the closest wrong one or quit. Add “Other”, “None of these” or “Not applicable” when they are real answers.
Overlapping rangesRanges like 10–20 and 20–30 put a person who answers 20 in two groups. Make every range end before the next begins.
Option order (primacy)In long lists, the first options are chosen more often simply because people read them first. Randomize the order of options that have no natural order.
Forced answersMaking every question required pushes people to guess or click anything. Require only what the survey cannot do without, and offer “Prefer not to say” for sensitive questions.

Biased survey questions FAQ

What is a biased survey question?

A biased survey question is worded or structured so that it pushes people toward a particular answer, or makes some answers harder to give. Common examples are leading questions (“How much did you love…”), loaded questions with a hidden assumption, double-barreled questions that ask two things at once, and unbalanced answer scales.

What is an example of a leading survey question?

“How much did you enjoy our excellent service?” is leading because “enjoy” and “excellent” suggest a positive answer. A neutral version is “How would you rate our service?” with a balanced scale from Very poor to Very good.

What makes a survey question bad?

A bad survey question is one that respondents can’t answer accurately: it is vague, asks two things at once, assumes something that isn’t true for everyone, uses jargon, or offers answer options that are incomplete, overlapping or unbalanced. The result is data that describes your wording rather than your respondents.

What are examples of confusing survey questions?

Confusing questions include double negatives (“Do you disagree that the office should not close?”), questions with two time frames, questions without a unit (“How much do you spend?”), hypothetical questions and long questions with several conditions. Each can be fixed by asking one specific thing in plain words.

What is acquiescence bias in surveys?

Acquiescence bias is the tendency of respondents to agree with statements regardless of their content. It inflates agreement on agree/disagree items. You can reduce it by asking item-specific questions (“How useful was the training?”) with a labeled scale instead of only agree/disagree statements.

How does question order cause bias?

Earlier questions shape how people read later ones. Asking about problems with delivery before an overall satisfaction question tends to pull the overall rating toward the delivery experience. Ask general questions before specific ones on the same topic, and randomize answer options that have no natural order.

How can I check my survey for bias?

Read every question against a checklist (no hinting adjectives, one topic per question, no hidden assumptions, specific time frames, balanced and complete options), then ask two or three people outside the project to complete it on a phone while thinking aloud.

Can I copy these examples?

Yes. You can copy and adapt any example on this page, both the biased versions for training and the fixed versions for your own surveys.

Neutral questions. Honest answers.

Create your first form in minutes with the free Early plan. No credit card required.