Rewriting Bad User Interview Questions: 12 Before-and-After Examples cover image
Photo by Mina Rad on Unsplash
Back to blog
Tutorial

Rewriting Bad User Interview Questions: 12 Before-and-After Examples

Twelve faults that weaken user interview questions, from leading and hypothetical to too broad, each with a before and after rewrite, a worked example guide and an eight-point check to run before your sessions.

Koji

Koji Team

Research · · 9 min read

You already have a discussion guide. Some of its questions are going to get you polite, vague or made-up answers, and you have a few days before the sessions start. This post is about fixing that: twelve common faults in interview questions, each with a before and after rewrite and why the new version works. For the full method of writing questions from scratch, read our reference guide, How to Write Great Interview Questions.

The short version: almost every bad interview question fails in one of three ways. It hands the participant an answer (leading, loaded, presupposing), it asks them to imagine instead of remember (hypothetical, future, "usually"), or it asks two things or nothing in particular (double-barreled, too broad, yes/no). The fix is nearly always the same move: ask about one specific thing that already happened, in neutral words.

If you want to try this on your own guide now, paste your questions into a Koji study and ask the research assistant to rewrite any question that is leading or hypothetical. It changes the question you name and tells you what it changed, so you can compare before and after.

Why small wording changes matter

Wording changes answers more than most teams expect. In Loftus and Palmer's classic experiment, people who watched the same car crash estimated the speed at 40.8 mph on average when asked how fast the cars were going when they "smashed" into each other, and 34.0 mph when the verb was "hit". A week later, 32% of the "smashed" group remembered seeing broken glass, against 14% of the "hit" group. There was no broken glass in the film (Journal of Verbal Learning and Verbal Behavior, 1974).

A follow-up by Loftus and Zanni found that asking "Did you see the broken headlight?" made people about twice as likely to say yes (17%) as asking "Did you see a broken headlight?" (7%), again when there was none (Bulletin of the Psychonomic Society, 1975).

Surveys show the same effect at scale. In January 2003, Pew Research Center found 68% of Americans favoured military action in Iraq to end Saddam Hussein's rule. When the question added "even if it meant that U.S. forces might suffer thousands of casualties", support fell to 43% (Pew Research Center, 2003). Same topic, one added clause, 25 points.

An interview is more forgiving than a survey, because you can follow up. But the first question in each section sets the frame for everything after it, so it is worth getting right.

Twelve rewrites

Table of twelve interview question rewrites: the fault, the original question and a better version for each, from leading and hypothetical to too broad.
Twelve common faults and the rewrite for each. Keep it next to your guide when you edit. · Download or share

1. Leading

Before: "How frustrating is it when exports fail?"

After: "Tell me about the last time you exported a report. What happened?"

The original assumes exports fail and that it is frustrating. The rewrite lets them tell you whether either is true.

2. Hypothetical

Before: "Would you use a feature that summarised your calls?"

After: "What did you do with your notes after your last call?"

People are poor at predicting what they would do and kind about your ideas. What they did last week is a fact.

3. Future price

Before: "Would you pay a monthly fee for this?"

After: "What do you pay for today to deal with this, if anything?"

A yes to a hypothetical price costs nothing. What they already spend, in money or time, is the real anchor. If you need price evidence, use a pricing method built for it, not one interview question.

4. Double-barreled

Before: "How do you plan and track your sprints?"

After: "How do you plan a sprint?" Then, separately: "How do you keep track once it starts?"

Two questions in one get one answer, and you will not know which part it answered.

5. Presupposing

Before: "What do you dislike about the dashboard?"

After: "When did you last open the dashboard? What were you looking for?"

The original assumes they use it and dislike something. As Loftus and Zanni showed, people go along with what a question takes for granted. Our guide to presupposition in interview questions goes deeper.

6. Loaded words

Before: "How much time do you waste on manual reporting?"

After: "How long did your last report take, from start to finish?"

"Waste" and "manual" tell them what you think. Ask for the number and let them judge it.

7. Closed yes or no

Before: "Is onboarding easy?"

After: "Walk me through your first day with the product."

A yes or no ends the topic. A walk-through gives you the steps, and the friction shows up on its own.

8. "Usually"

Before: "How do you usually choose a supplier?"

After: "Tell me about the last supplier you chose. How did it go?"

"Usually" invites a tidy average that may never have happened. One real instance has detail you can trust.

9. Asking for the solution

Before: "What features would you like us to add?"

After: "What did you have to do outside the product last week to get the job done?"

Participants are experts on their problems, not on your roadmap. Workarounds show you the gap without asking them to design the fix.

10. Too broad

Before: "Tell me about your workflow."

After: "Tell me about the last time you handed a project to a client."

A broad question gets a broad answer. Pick the moment you care about and ask about one instance of it.

11. Jargon

Before: "How do you handle stakeholder alignment in your discovery cadence?"

After: "Before you start a new project, who do you talk to, and about what?"

If they would not say it out loud, do not ask it. Their own words are also data: note what they call things.

12. "Why" that sounds like blame

Before: "Why didn't you finish setting it up?"

After: "What was happening when you stopped setting it up?"

"Why didn't you" puts people on the defensive. Asking about the moment gets the reason without the excuse.

A worked example: one guide, before and after

Here is a short guide for a study on why trial users go quiet, as a team first wrote it:

  1. "Did you like the product during your trial?"
  2. "Why didn't you upgrade?"
  3. "Would more templates have helped?"
  4. "How do you usually manage your projects and team?"
  5. "What would make you come back?"

And rewritten:

  1. "What were you hoping to get done when you signed up?"
  2. "Tell me about the last time you logged in. What were you trying to do?"
  3. "What did you use instead, after the trial ended?"
  4. "Tell me about the last project you started. How did you set it up?"
  5. "Is there anything I should have asked about your trial and didn't?"

Each original question had at least one fault from the list: closed (1), blame (2), hypothetical and leading (3), double-barreled and "usually" (4), future (5). Every rewrite asks about one real moment, and none of them mentions a feature.

Rewriting follow-ups

Your probes need the same care as your main questions. "So that was annoying?" is a leading probe. "And how did you feel about that?" can push people into an emotion they did not have. Short, neutral follow-ups work best: "What happened next?", "Can you give me an example?", "What did you do then?", "Tell me more about that." Our guide to probing and follow-up questions has more examples, and avoiding leading questions covers the survey side.

A last check before you run the guide

Read every question out loud, then run it against this list.

A researcher at a desk going through a printed interview guide with a pen, marking questions to rewrite before the sessions
A researcher at a desk going through a printed interview guide with a pen, marking questions to rewrite before the sessions

Photo by Aaron Burden on Unsplash

Checklist of eight checks for interview questions, from asking about one real past moment to using neutral words, one topic per question and no feature names.
Run every question in your guide through these eight checks before the first session · Download or share
  • Does it ask about one specific thing that already happened?
  • Is it one question, not two?
  • Could someone answer it honestly without agreeing with you?
  • Are there any loaded words (waste, struggle, frustrating, easy)?
  • Does it assume something you have not established?
  • Would the participant use these words themselves?
  • Does it avoid naming your feature or idea?
  • Can it be answered with more than yes or no?

Then do one practice run with a colleague, or a test interview, and listen for where the answers go flat. A flat answer usually points to the question before it.

Rewriting questions in Koji

In Koji, your guide lives in the study's questions editor, where each question has its own follow-up setting. You can edit any question by hand, or ask the research assistant to change one ("make question 3 about the last time they did it, not what they would do"); it changes the question you name, leaves the rest of the guide alone and tells you what it changed.

Koji question editor showing open ended, 1 to 10 scale and single choice questions with follow-up settings
Each question in a Koji guide has its own follow-up setting, so the AI interviewer probes where you want depth

When the study runs, Koji's AI interviewer asks your questions and follows up on what each person actually says, with neutral probes, in voice or text. You can preview the interview yourself before it goes live, which is the quickest way to hear a question that does not land. Each interview then gets a quality rating, so answers that went nowhere are easy to spot.

Start a free study and paste in the guide you are working on, or read how Koji's AI follow-up probing works first.

Run your first AI-moderated study in 10 minutes

10 free credits on signup. No credit card required.

GDPR compliantEU or US data residencyNo AI training on your data
Koji

Koji Team

Research

Share this article

Keep reading