If an AI assistant drafted your requirements, someone still has to check them against the people who will live with them. This playbook gives you a way to do that in a week: run short AI-moderated read-back interviews, where each stakeholder rules on a handful of specific requirements in their own words. By the end you can walk into sign-off with a verdict for every requirement, the people behind it and the quotes that explain it.
Why validating AI-drafted requirements matters now
Drafting got cheap. Checking did not. IIBA's 2026 Global State of Business Analysis report draws on 2,120 professionals in 122 countries, and IIBA's summary reports that 69% say AI is having a positive effect on their careers. IIBA's own practitioner writing on AI in validation is just as clear that the analyst stays central, using judgment, context and stakeholder understanding to guide and validate AI output.
Photo by Vitaly Gariev on Unsplash
The draft is fast. The evidence that it is right is not. A 2025 study by Quattrocchi and colleagues had ten language models play the customer in interviews and write user stories. The stories matched human ones on coverage and style, but showed lower diversity and met the acceptance quality criteria less often, whatever the model size. A 2026 survey of AI test generation from requirements found that hallucination control is weakly addressed, with four of eight representative studies rated Low.
There is a second trap, and it is a human one. In a 2026 REFSQ study, 26 participants rated LLM revisions of the 130 requirement statements they had written higher on alignment, readability, reasoning and unambiguity. The authors keep stakeholders in the validation loop. My reading of it: polished text is easy to approve. A clean draft invites a quick "looks fine", and "looks fine" is not the same as "right".
Requirements were already where projects bleed. PMI's 2014 Pulse of the Profession found that 37% of organizations named inaccurate requirements as the primary reason for project failure. PMI's 2026 Pulse says about one in three complex projects fails, roughly twice the 13% rate for projects overall.
The cost of skipping validation: the draft is signed on the strength of a meeting where two people spoke, and the first honest review is the user acceptance test. By then the wrong requirement is code.
What good looks like
Validation is not elicitation with a document attached. In elicitation you ask people what they need. In validation you show people a specific claim and ask whether it is true for them. I call the method read-back validation: each person reads back a small set of requirements, rules on each one, and says what is missing.
Package requirements in sets of six. Group the draft by who it affects: approvals, data entry, reporting, exceptions. Give each interviewee one set of at most six requirements, plus one open question about what is missing. Past six, people start agreeing to finish. A set takes 10 to 12 minutes in text.
Four answers per requirement. Each requirement gets one question with four options: right as written, right but needs changing, wrong, or not something I can judge. "Not something I can judge" matters. Without it, people guess to be polite, and the guess goes in your tally as a confirmation.
Cover every requirement three times, from two roles. Whoever commissioned the work, plus at least two people who do it daily. A requirement checked only by its sponsor is checked by the person least likely to hit the exception.
Plant one decoy per set. Alter one requirement so it is subtly wrong, for example a threshold of 5,000 where the draft's source says 500. Tell everyone in the invitation that some statements may be wrong and that you want to hear it. If someone confirms the decoy, discount their other confirmations. This is a practitioner's check, not a published method, and it works because it costs one question.
Decision rule for sign-off. Treat a requirement as validated when at least three people from at least two roles rule on it and at least 80% say "right as written". Take it to a short workshop when two or more people say "wrong" or "needs changing", or when two roles disagree. Anything with fewer than three verdicts is not validated yet. These thresholds are rules of thumb from running this kind of review, not a standard.
Common mistakes:
- Sending the whole specification. Nobody reads 90 requirements, and "looks fine" comes back from all of them.
- Asking "do you agree?" People agree. Ask "is this how you do it today?" and anchor on the last time they did.
- Showing the AI's reasoning or source. It anchors people. Show the requirement, not where it came from.
- Validating only with managers. They confirm policy. The exception lives with the person who covers the desk.
- Counting a confirmation without a role next to it.
- Treating one dissent as noise. One "wrong" from the person who runs month end is worth more than five confirmations from people who never see it.
How to set it up in Koji, step by step
Text interviews work on every plan, including Free. Voice interviews need the Interviews plan or Enterprise. For read-back, text is the better default: people look at the exact wording of a requirement while they answer.
- Start from the template. In the dashboard's template gallery, open the Business Analysts templates and pick "Stakeholder Requirements Gathering". Keep its brief structure and change the question, or type your question on the home screen and let Koji draft a brief.
- Rewrite the brief. The problem statement becomes: "Which of these requirements are right as written, which need changing, and what is missing, according to the people who use the process?" Name the decision: what goes to baseline sign-off.
- Upload the draft as a context file. Add the full requirements draft and any source documents (PDF, TXT, JSON or MD, up to 25 MB each) so the interviewer understands the vocabulary and asks sharper follow-ups. The participant does not see the file, so the requirement wording itself goes in each question.
- Build one question per requirement. Use single choice with your four answers and paste the requirement text into the question. Set follow-up probing to 2 on the questions where you most need a reason. The interviewer can ask zero to three follow-ups per question; see how AI follow-up probing works.
- Add one open question about gaps, then preview. Run a preview as yourself. It is saved with its own analysis but not counted as a response.
- Invite and read. Share the interview link per stakeholder group, or import a CSV for personalised links on the Interviews plan or higher. Set a response limit if you want a cap.
Copy this interview guide
Paste this brief into the "What do you want to research?" box:
Show stakeholders six specific requirements from a draft written with AI help. For each one, find out whether it is right as written, needs changing or is wrong, why, and what is missing, so the team can sign off a baseline with evidence from the people who do the work.
Opening line: "I'm checking a draft of requirements for [project]. Some statements may be wrong, and I want to hear which. Your answers go to the project team, and this is not an assessment of you."
Questions 1 to 6 follow the same pattern, one per requirement:
- (Single choice) "Requirement 1: [paste requirement]. Is this how it works for you today? Right as written / Right but needs changing / Wrong / Not something I can judge." Probe: "Tell me about the last time this came up."
- (Single choice) Same format for requirement 2. Probe: "What would you change in the wording?"
- (Single choice) Requirement 3, plus the decoy in one of the six. Probe: "What makes you say that?"
- (Single choice) Requirement 4. Probe: "Who else would see it differently?"
- (Single choice) Requirement 5. Probe: "What happens if this is not in place on day one?"
- (Single choice) Requirement 6. Probe: "How would you know it was working?"
- (Open) "What is missing from these six that you would expect to see?" Probe: "When did that last cause trouble?"
- (1 to 10 scale) "How confident are you that these six cover what your team needs on day one? 1 is not at all, 10 is completely." Probe: "What would move your score up two points?"
Closing question: "Is there anything about how you work that this draft would get wrong?"
Add this as a checklist for the team: freeze the draft, tag each requirement with its source, split into sets of six, plant one decoy per set, invite three people per requirement from two roles, close in five working days.
What you get
Each interview is analysed on its own, with a quality rating, key findings and pain points. The report aggregates your single choice questions into a distribution per requirement and pulls the open answers into themes with the share of participants behind them, and each finding links to supporting responses. See how to analyse AI-moderated interview results.
The next day you build the verdict table: one row per requirement, the answers by role, the decision (validated, amend, workshop), and the quote that explains it. In the sign-off meeting you stop arguing about opinions. You say "14 of 18 are validated, three are in a 30-minute workshop on Thursday, and one was wrong in a way that would have broken month end." You also get the missing-requirements list straight from the gap question, with the participant's words.
The same playbook elsewhere
The same read-back works whenever a plan was written without the people it affects. Agencies use it to test a proposal against each client team; see new-client stakeholder discovery. Founders use it to hear a pilot's end users before the sponsor decides; see enterprise pilot interviews. For the step before the draft, AI agent workflow triage shows how to find what is worth drafting requirements for. HR can run it on a draft policy before it ships.
Why Koji for requirements validation
Pricing for these alternatives varies and is not always public, so the table compares what each method gives you.
- People answer alone. No sponsor in the room and no loudest voice. A "wrong" is easier to say to an interviewer who does not report to anyone.
- Every "wrong" gets a "why". The interviewer follows up on each answer, up to three times, so you get the reason and the last real case, not a bare disagreement.
- Same wording for everyone. One guide means a difference in verdicts is a difference in the work, not in the interviewer.
- It fits calendars. Participants answer when they are free, so twenty stakeholders do not need twenty slots. See asynchronous interviews.
- A tally you can defend. Single choice answers become a count per requirement, with the quotes behind them, which is what a sponsor asks for when you change a requirement they liked.
Questions you might have about AI interviews
Will stakeholders tell an AI that a requirement is wrong? Often more readily than they would tell the person who wrote it. Say in the invitation that some statements may be wrong, who reads the results and that this is not an assessment. For evidence on how people respond, read will customers open up to an AI interviewer.
Can it handle a precise requirement without paraphrasing it? The requirement text is in the question you wrote, so every participant sees the same wording. Check that in a preview before launch.
What happens to what people say? Tell participants in the invitation how their answers are used, and read the AI interview data privacy and security guide before you invite anyone. Use guardrails to exclude topics you do not want covered. Only conversations that pass the quality gate consume credits.
When Koji may not be the right fit
If the draft has ten requirements and four stakeholders who sit in one room, a 45-minute walkthrough does the job and is faster. If a requirement is a legal or regulatory obligation, it is not a matter of opinion and interviews cannot validate it. If you need to see people use a prototype, put the prototype in front of them. Koji is for the middle: dozens of requirements, a dozen or more stakeholders across roles, and a baseline that needs evidence.
Metrics to track
- Share of requirements validated against the three-people, two-roles rule
- Requirements amended or rejected after read-back, and the share of the draft they represent
- Gaps found per set of six, and how many became requirements
- Decoys confirmed (a rubber-stamping signal)
- Change requests raised after sign-off, compared with your last project
FAQ
How many requirements per interview? Six at most, plus one gap question. Beyond that, people agree to finish.
How many people do I need per requirement? At least three, from at least two roles.
Is it fair to plant a decoy? Yes, if you tell everyone in the invitation that some statements may be wrong. It measures rubber-stamping, not trickery.
Can I run this on the Free plan? Yes, with text interviews and the template. Voice and CSV import of participants need a higher plan.
Does this replace a requirements workshop? No. It tells you which requirements need one. Take the disputed ones there.
What does it cost in credits? A text interview is 1 credit and a voice interview is 3 credits. Only conversations that pass the quality check consume credits.
Start your read-back
Open the Business Analysts templates, pick Stakeholder Requirements Gathering and paste the brief above. More about the role is on the Koji page for business analysts, every playbook for analysts is in the use case library, and the general method is in the requirements gathering interviews guide.
