{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-05-02T15:31:56.578Z"},"content":[{"type":"blog","id":"8837e97d-2e1a-49e1-8f29-98d4079d71c8","slug":"koji-vs-synthetic-users-2026","title":"Koji vs Synthetic Users: Real Customer Voices vs AI Personas (2026)","url":"https://www.koji.so/blog/koji-vs-synthetic-users-2026","summary":"Comparison of Koji and Synthetic Users for AI-powered research. Synthetic Users generates LLM-based personas; Koji uses AI to moderate interviews with real customers. Synthetic data cannot be validated against external ground truth, tends to praise concepts uncritically, and reflects LLM training-data bias. Koji offers the same speed advantage with real customer voices, six structured question types, voice and text interviews, and quote-evidenced thematic analysis.","content":"\n# Koji vs Synthetic Users: Real Customer Voices vs AI Personas (2026)\n\nSynthetic Users is one of the most-discussed names in AI research right now. The pitch is irresistible: skip recruitment, skip scheduling, skip the messy work of getting real humans on a call. Generate AI personas, \"interview\" them, get a report in minutes.\n\nIt is fast. It is cheap. And for some narrow use cases — early hypothesis generation, desk research, identifying what to investigate next — it can save real time.\n\nBut it is also being marketed as a replacement for talking to actual customers, and that is where the conversation gets serious. The Nielsen Norman Group, ACM Interactions, and a growing body of academic literature have all raised the same fundamental concern: **synthetic users cannot be validated, because they have no connection to observed reality.** Insight built on synthetic data is an assumption built on top of other assumptions, all the way down.\n\nKoji takes the opposite approach. The AI is the interviewer, but the participants are real customers — your customers, recruited through your own channels or Koji's panel, talking about their actual experience. Same speed advantages, none of the validity concerns.\n\nThis guide covers exactly where Synthetic Users fits, where it does not, and how Koji handles the same job differently.\n\n---\n\n## What Synthetic Users Actually Does\n\nSynthetic Users is an AI platform that generates synthetic participants — LLM-based agents calibrated to demographic, psychographic, and OCEAN (Big Five personality) profiles. You describe the participant you want to \"interview,\" set up a discussion guide, and the platform runs an AI-versus-AI conversation: one agent asks questions, another agent plays the participant, a third agent critiques.\n\nThe output looks like a real research transcript. There is a participant name, a personality profile, follow-up probing, and a thematic summary. For €2–€27 per interview (with a small RAG add-on if you want the agents to reference your own documents), you can generate dozens of these in an afternoon.\n\n**Where this is genuinely useful:**\n\n- **Discussion guide stress-testing.** Before you spend money recruiting real people, run your guide against a few synthetic personas to find leading questions or confusing wording.\n- **Hypothesis brainstorming.** Generating a wide funnel of \"what might users think?\" hypotheses before deciding which ones to test with real people.\n- **Internal alignment exercises.** When stakeholders cannot agree on who the user is, synthetic personas can structure the disagreement before you commit to recruiting.\n- **Desk research at scale.** Sketching market segments quickly when no real-data alternative exists.\n\nThese are real jobs and synthetic users can help with them. The problem starts when teams use synthetic personas as if they were participants — making roadmap, pricing, or positioning decisions on data that has no grounding in actual customer behavior.\n\n---\n\n## The Validity Problem\n\nThe critique of synthetic users is not \"AI is bad.\" The critique is specific and methodologically rigorous, and it deserves a clear hearing.\n\n### Synthetic users praise everything\n\nA widely-cited finding from research published in 2025–2026 is that synthetic users **care about everything equally** and **praise concepts without criticism.** Real customers have sharp preferences, surprising blind spots, and load-bearing objections. Synthetic agents are trained to be helpful and fluent, which means they tend to validate whatever you put in front of them. For concept testing or feature prioritization, this is the opposite of what you need.\n\n### LLMs reflect the dominant voices in their training data\n\nLLMs are trained predominantly on English-speaking, affluent, tech-literate text. Marginalized perspectives, non-Western users, low-income segments, and edge-case users are systematically underrepresented. A synthetic persona of a \"rural farmer in Indonesia\" is not actually a rural farmer in Indonesia — it is what the model statistically associates with that label, which is closer to how Western media depicts that persona than how that person would actually answer.\n\n### No theory of mind, no preferences, no memory\n\nLarge language models convert words into vectors and produce statistically probable next tokens. They do not know, want, remember, or believe anything. There is no stable preference structure underneath the persona — just patterns. Two \"interviews\" with the same synthetic persona, on the same topic, will often produce contradictory answers because there is no underlying participant who actually exists.\n\n### Cannot be validated\n\nThis is the crux. A real interview can be wrong but it can also be checked — against analytics, against support tickets, against the participant's actual behavior in your product. A synthetic interview produces a transcript that *sounds* plausible but has no external referent. You cannot fact-check what a non-existent person said. There is no ground truth.\n\nThe ACM Interactions essay summarized it bluntly: synthetic users are assumptions built on assumptions. If you make decisions based on them, you are making decisions based on what your LLM thinks plausible-sounding people would say — which is rarely what your actual customers will do.\n\n---\n\n## What Koji Does Differently\n\nKoji is also AI-moderated. The AI conducts the interview, asks follow-up questions, and synthesizes themes. The difference is that Koji interviews **real people** — the actual customers and prospects you want to understand — at the same speed synthetic platforms claim.\n\nHere is how Koji solves the bottleneck synthetic users tries to solve:\n\n- **Always-on AI interviewer.** Send a link, the AI conducts the interview whenever the participant is available. Asynchronous, voice or text, no scheduling required. See [always-on user interviews](/docs/always-on-user-interviews-24-7-ai-moderator).\n- **Automatic probing.** When a participant says something interesting, the AI asks the follow-up. Not a synthetic follow-up — a real one, to a real person. Detailed in the [AI probing guide](/docs/ai-probing-guide).\n- **Six structured question types.** Open-ended, scale, single choice, multiple choice, ranking, and yes/no — all in the same study. See the [structured questions guide](/docs/structured-questions-guide).\n- **Automatic thematic analysis.** Across all completed interviews, with quote evidence. Real quotes from real customers. See the [AI transcript analysis guide](/docs/ai-transcript-analysis-guide).\n- **Voice and text both supported.** Voice interviews capture nuance that text loses; text interviews work for participants who prefer typing. Configured in [setting up voice interviews](/docs/setting-up-voice-interviews).\n- **GDPR-compliant consent collection.** Built into every study, see [intake forms and consent](/docs/intake-forms-and-consent).\n\nWhat you get is the speed of synthetic research with the validity of real research.\n\n---\n\n## Side-by-Side Comparison\n\n| Capability | Synthetic Users | Koji |\n|---|---|---|\n| Who are the participants? | LLM-generated AI personas | Real customers (yours or recruited) |\n| Can findings be validated? | No external ground truth | Yes — quotes, transcripts, real participants |\n| Probing follow-up questions | Yes (AI to AI) | Yes (AI to real human) |\n| Voice interviews | Text only | Voice and text |\n| Thematic analysis | Yes, on synthetic data | Yes, on real customer data |\n| Question types | Free-form | 6 structured types + open-ended |\n| Risk of training-data bias | High — reflects LLM's dominant voices | None — actual respondent voices |\n| Concept testing reliability | Low — synthetic personas tend to praise everything | High — real users push back |\n| Pricing | ~€2–€27 per synthetic interview | Free to start; from €29/mo (29 credits) |\n| Audit trail for stakeholders | Synthetic transcripts | Real transcripts, real participants, real quotes |\n\n---\n\n## When Synthetic Users Is the Right Tool\n\nTo be fair: there are jobs where synthetic users is the better choice.\n\n**Use Synthetic Users when:**\n\n- You are stress-testing a discussion guide before recruiting real participants\n- You are brainstorming hypotheses that you will later validate with real users\n- You need to sketch personas for an internal alignment workshop where the goal is structured disagreement, not data\n- You are doing desk research in a domain where real participants are genuinely unreachable in your timeframe\n- The decision the research informs is reversible and low-stakes\n\nSynthetic Users honestly has a place in the modern researcher's toolkit — as a complement to real research, not a substitute for it.\n\n## When Koji Is the Right Tool\n\n**Use Koji when:**\n\n- The decision is irreversible or high-stakes (roadmap, pricing, positioning, hiring, fundraising)\n- You need to understand *why* real customers behave the way they do\n- Stakeholders need quote evidence from real people in the report\n- You are doing churn analysis, win-loss analysis, or concept validation\n- The audit trail of the research matters (board, investors, regulators)\n- You want the speed of AI moderation without the validity tradeoff\n\nThe rule of thumb: if a real decision will be made on the back of the research, the participants need to be real.\n\n---\n\n## What This Looks Like in Practice\n\n**Scenario: Validating a new pricing tier.**\n\nWith Synthetic Users, you spin up 30 synthetic \"SMB founders\" and ask them about willingness to pay for a Pro tier at $99/mo. They respond fluently. Most express interest. The report says SMB founders are receptive to the price point.\n\nYou launch the tier. Conversion is 0.4% of trials. The real customers, when interviewed afterward, say the price was the wrong shape entirely — they would have paid annually, but not monthly, and the value was unclear without a feature you had not even highlighted.\n\nWith Koji, you import 30 real trial users (or open a public link) and run a [pricing research interview](/docs/pricing-research-interviews) with [willingness-to-pay questions](/docs/willingness-to-pay-interview-template). The AI probes for the *shape* of pricing (annual vs monthly, per-seat vs flat) not just the number. Themes surface: real customers ask for an annual option, real customers fixate on a feature you had not foregrounded. You ship the right tier on the first try.\n\nSpeed is the same. Validity is not.\n\n---\n\n## How Teams Use Both Tools\n\nThe sophisticated approach: use synthetic and real research in series, not in parallel.\n\n1. **Synthetic phase (1 day).** Generate hypotheses about what real customers might say. Stress-test your discussion guide. Identify leading questions.\n2. **Real research with Koji (3–5 days).** Run the refined guide against real customers. AI-moderated, asynchronous, voice or text. Get themes with real quote evidence.\n3. **Decision (week 2).** Make the call on real data, with synthetic data only used to inform what you investigated.\n\nThis is how the methodologically rigorous teams at companies that take research seriously are using synthetic tools today: as scaffolding, not as the building.\n\n---\n\n## Start a Real-Customer Study with Koji\n\n[Koji](https://koji.so) is free to start. Import your customers from CSV, share a public link, or use Koji's panel. The AI conducts every interview asynchronously, probes follow-ups automatically, and delivers a thematic report with quote evidence — from real people, in days.\n\n**[Start free →](https://koji.so/signup)**\n\n**Related reading:** [AI-moderated vs human-moderated interviews](/blog/ai-moderated-vs-human-moderated-interviews) · [How to run AI-powered customer interviews at scale](/blog/how-to-run-ai-powered-customer-interviews-at-scale) · [Why AI interviewers are the future of customer research](/blog/why-ai-interviewers-are-the-future-of-customer-research) · [Best AI customer interview tools in 2026](/blog/best-ai-customer-interview-tools-2026) · [Customer discovery: the ultimate guide for startups](/blog/customer-discovery-the-ultimate-guide-for-startups-2026)\n","category":"Research","lastModified":"2026-05-01T03:16:29.619073+00:00","metaTitle":"Koji vs Synthetic Users: Real Customers vs AI Personas (2026)","metaDescription":"Synthetic Users generates AI personas. Koji interviews real customers with AI moderation. Here is exactly when each is appropriate — and why insight built on synthetic data carries validity risk synthetic vendors rarely discuss.","keywords":["koji vs synthetic users","synthetic users alternatives","AI personas vs real users","synthetic user research limitations","AI moderated interviews 2026"],"aiSummary":"Comparison of Koji and Synthetic Users for AI-powered research. Synthetic Users generates LLM-based personas; Koji uses AI to moderate interviews with real customers. Synthetic data cannot be validated against external ground truth, tends to praise concepts uncritically, and reflects LLM training-data bias. Koji offers the same speed advantage with real customer voices, six structured question types, voice and text interviews, and quote-evidenced thematic analysis.","aiKeywords":["synthetic users","AI personas","AI research platforms","customer interviews AI","user research validity","AI moderated interviews"],"aiContentType":"comparison","faqItems":[{"answer":"For most decisions, no. Research from Nielsen Norman Group, ACM Interactions, and academic studies in 2025–2026 has documented that synthetic users tend to praise concepts without criticism, reflect LLM training-data bias, and cannot be validated against external ground truth. They are useful for hypothesis generation and discussion guide stress-testing, but real decisions should be informed by real customer voices.","question":"Are synthetic users a valid replacement for real customer interviews?"},{"answer":"Synthetic Users generates AI personas and runs simulated AI-versus-AI interviews. Koji uses AI to moderate interviews with real customers — your actual users, recruited via CSV import, public links, or Koji panel. Both are fast and AI-driven; only Koji produces validatable data based on real customer experience.","question":"What is the difference between Synthetic Users and Koji?"},{"answer":"Synthetic Users charges roughly €2–€27 per synthetic interview, plus an additional small fee for RAG (Retrieval-Augmented Generation) to add your own documents. Koji is free to start, with paid plans from €29/month (Insights) or €79/month (Interviews) including unlimited studies and voice interview capabilities.","question":"How much do Synthetic Users and Koji cost?"},{"answer":"Yes — and this is the methodologically sound approach. Use Synthetic Users to brainstorm hypotheses and stress-test your discussion guide before recruiting. Then use Koji to run the refined guide against real customers and make decisions on real data with quote evidence.","question":"Can I use synthetic users and Koji together?"},{"answer":"No. The AI moderator is consistent across every session, does not signal preferred answers, does not get tired or distracted, and probes follow-up questions based purely on what the participant said. This eliminates moderator bias while preserving real participant voices.","question":"Do AI-moderated interviews on Koji introduce bias the way human interviewers do?"},{"answer":"Anything where the decision is high-stakes, irreversible, or needs an audit trail: pricing research, churn analysis, win-loss interviews, concept validation for shipped features, board-facing strategic research, regulated-industry research, and any work where stakeholders need quote evidence from real people.","question":"What kinds of research are real-customer interviews better for than synthetic ones?"}],"relatedTopics":["synthetic users alternatives","AI research platforms","AI personas vs real users","customer research AI 2026","synthetic data validity","AI moderated interviews"]}],"pagination":{"total":1,"returned":1,"offset":0}}