All frameworks and cheat sheets

Process

User research for AI products in 5 steps

Process with five steps of user research for AI products: generative research, concept tests, Wizard-of-Oz prototypes, a hallucination beta, ongoing research.
Each step answers what a model benchmark cannot: will people trust it, and what happens when it is wrong.

A five-step research plan for teams shipping generative AI features: generative interviews, concept tests, Wizard-of-Oz prototypes, a beta that measures hallucination tolerance, and research built into the product. Use it to decide what to test at each stage, from the first idea to weekly model changes. Koji's AI-moderated interviews cover the interview work in each step.

Where this appears

What's in this process

  1. Generative research: Learn the job users do by hand today and what worries them about AI doing it.
  2. Concept testing: Check comprehension, desirability and the accuracy users would need, before you ship the model.
  3. Wizard-of-Oz prototype: A person plays the AI. Watch whether users notice errors and how they recover.
  4. Hallucination tolerance beta: With 100+ users on the live model, track discovery rate, recovery cost and forgiveness.
  5. Continuous embedded research: In-product prompts and monthly interviews, because models and prompts keep changing.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/ai-product-user-research-five-steps"><img src="https://www.koji.so/visuals/ai-product-user-research-five-steps/ai-product-user-research-five-steps.png" alt="Process with five steps of user research for AI products: generative research, concept tests, Wizard-of-Oz prototypes, a hallucination beta, ongoing research." width="1200" height="739" /></a>
<p>Source: <a href="https://www.koji.so/visuals/ai-product-user-research-five-steps">User research for AI products in 5 steps</a>, Koji</p>