{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-21T16:51:25.209Z"},"content":[{"type":"documentation","id":"32b30ec0-acf1-4a27-89cd-58f39e0668cb","slug":"survey-evidence-court-daubert","title":"Survey Evidence in Court: Daubert, FRE 702, and Research That Survives Cross-Examination","url":"https://www.koji.so/docs/survey-evidence-court-daubert","summary":"The bar survey evidence must clear in federal court is competent research with the methodology documented, which makes it a useful internal quality standard. This guide covers FRE 703 and FRE 702 as amended effective 1 December 2023, the Reference Manual on Scientific Evidence Fourth Edition (2025) and its Reference Guide on Survey Research, the five-element reporting record, and a table mapping each common attack to the design choice that defeats it. Key findings: representativeness matters more than response rate, an explicit do-not-know option shifts selection by 20 to 25 percent, agree or disagree formats carry a 10 percent acquiescence inflation, and pretests run 25 to 75 respondents.","content":"**The bar a survey must clear to be admitted in federal court is not exotic. It is competent research with the methodology written down.** That makes it unusually useful as an internal quality standard, because unlike \"did the stakeholders find this convincing\", it is specific, published, and adversarial by design. If you want a single question that reliably separates research that will hold up from research that will not, use this one: could this survive someone whose job is to destroy it?\n\nMost product research would not, and the reason is almost always the same missing design choice. We will get to it.\n\n## The legal frame, briefly\n\nTwo rules do the work. Federal Rule of Evidence 703 settled the old objections to surveys, that they rest on sampling and that they are hearsay, by redirecting attention to the validity of the techniques employed; the inquiry is whether the facts or data are of a type reasonably relied upon by experts in the field. Federal Rule of Evidence 702, as amended effective 1 December 2023, then governs the expert who presents them. It now states explicitly that the proponent must demonstrate to the court that it is **more likely than not** that the testimony rests on sufficient facts or data, is the product of reliable principles and methods, and that the expert's opinion reflects a reliable application of those principles and methods to the facts of the case. The 2023 amendment was made because courts had been applying that burden too loosely. *Daubert v. Merrell Dow Pharmaceuticals, Inc.*, 509 U.S. 579 (1993), remains the source of the gatekeeping role.\n\nThe practical guidance judges actually consult is the Reference Manual on Scientific Evidence, whose **Fourth Edition was published by the National Academies Press on 31 December 2025**. Its Reference Guide on Survey Research, by Shari Seidman Diamond, Matthew Kugler and James N. Druckman, is the closest thing that exists to an official checklist for evaluating a survey.\n\n## The five things you must be able to describe\n\nThe Reference Guide frames adequate reporting around five elements. Treat this as the minimum record for any study you might ever need to defend.\n\n| Element | What it means | What to record at the time |\n| --- | --- | --- |\n| Population | A description of the population sampled, showing it was the relevant one for the question | Who the finding is about, and why that group and not a broader or narrower one |\n| Sample design | How the sample was drawn and why that design was appropriate | Recruitment source, screening criteria, quotas, probability or non-probability |\n| Response and representativeness | The response rate and the sample's ability to represent the target population | Invitations, starts, completes, screen-outs, and known differences from the population |\n| Respondent quality | Evidence that respondents were attentive and honest | Attention checks, duration distribution, duplicate and fraud controls |\n| Bias assessment | An evaluation of potential sources of bias in the answers | Question order, wording decisions, who knew the sponsor, and what was pretested |\n\nRecord these while the study runs. Reconstructing them afterwards is where most internal research quietly fails, and it is also where an opposing expert will start.\n\n## The attacks, and the design that defeats each\n\nThis is the table worth keeping. Every column-two entry is a decision made before fielding, not an argument made afterwards.\n\n| The attack | What it sounds like | The design choice that defeats it |\n| --- | --- | --- |\n| Leading question | You told them the answer you wanted | A **control cell** answering the identical question |\n| Guessing and noise | People will say yes to anything | A control cell, plus an explicit do-not-know option |\n| Acquiescence | Your agree or disagree scale inflated everything | Avoid agree or disagree formats; use neutral specific wording |\n| Order effects | The earlier question primed the later one | Ask general before specific; counterbalance and document the order |\n| Sponsor bias | Your client's logo was on the instrument | **Double-blind administration**: neither interviewer nor respondent knows the sponsor or purpose |\n| Wrong universe | You surveyed the wrong people | A population definition written down before recruiting |\n| Low response rate | Only a fraction replied, so it is meaningless | Evidence on whether non-response is systematic, not just a higher rate |\n| Undisclosed method | We cannot check what you did | The five-element record above |\n\n### The control group is a legal instrument\n\nIf you take one thing from this guide, take this. The Reference Guide makes the point crisply in the deception context: if respondents who saw the allegedly deceptive advertisement answer differently from respondents who saw a control advertisement, the difference cannot merely be the result of a leading question, **because both groups answered the same question**. Background noise, guessing, and even a poorly worded item should push both cells in the same direction, so the comparison survives objections that would destroy a single-cell study.\n\nThat is why so much product research is undefendable by construction. It runs one cell, produces a percentage, and has no answer at all when someone asks whether the number would have appeared anyway. Adding a control cell roughly doubles the sample and neutralises the three most common attacks at once. It is the highest-return design decision in applied research, and it is the same instrument that makes [dark patterns testing](/docs/dark-patterns-user-testing) and [clickwrap notice studies](/docs/clickwrap-vs-browsewrap-assent-research) meaningful rather than merely suggestive.\n\n## Three findings that should change how you run ordinary studies\n\nThe Reference Guide contains specific, quantified guidance that is directly useful outside litigation.\n\n**Stop panicking about response rate.** The guide states that surveys with varying response rates have produced surprisingly comparable results, and that generally the representativeness of a sample, regardless of response rate, matters much more for accurate inference than the response rate itself. What matters is whether non-response is associated with systematic differences that cannot be modelled or assessed. A high response rate from a skewed frame is worse than a low one from a good frame. This reframes a metric teams routinely over-weight, and it connects directly to [sampling bias](/docs/sampling-bias-research) and [survivorship bias](/docs/survivorship-bias-customer-research).\n\n**An explicit do-not-know option moves the numbers a lot.** Offering one commonly leads to a 20 to 25 percent increase in the proportion of respondents selecting it. That is not a rounding effect. Whether you offer it is a substantive design decision that should be made deliberately and reported, because a study without one has quietly converted uncertainty into opinion.\n\n**Agree or disagree formats carry a documented inflation effect of about 10 percent.** The guide describes acquiescence as the tendency to endorse any assertion regardless of content, and says the format only yields reasonable estimates when control groups, control questions or counterbalanced items are added. If your satisfaction tracker is built on agree or disagree items, some of your trend is format.\n\nTwo more worth internalising: pretests are typically run with **25 to 75 respondents** of the same type eligible for the main study, and courts have treated the absence of pretesting as a weakness. And double-blind administration means the instrument gives no clue to sponsorship or expected answers, down to details like not reversing the usual order of yes and no boxes on a key question.\n\n## How Koji fits\n\nMost of this checklist is about discipline and documentation, which is exactly where general-purpose survey tools leave you on your own. A Typeform or SurveyMonkey questionnaire will happily field an agree-or-disagree battery with no control cell, no do-not-know option and no record of why the population was defined as it was. Qualtrics gives you more control but still treats the qualitative half as a separate exercise, which means the explanations that make a finding legible arrive on a different timeline, if at all.\n\nKoji is built around the parts that make research defensible. All six structured question types, open_ended, scale, single_choice, multiple_choice, ranking and yes_no, live in one AI-moderated conversation, so the countable measures and the verbatim reasoning come from the same respondents in the same session rather than from two studies stitched together. The AI interviewer probes an ambiguous answer on the spot, which is how you distinguish a respondent who genuinely held a belief from one who was guessing, precisely the attentiveness evidence the fourth element of the record calls for. Interviews run by voice or text with no moderator to schedule, so a two-cell design with a real control is affordable rather than a luxury. And because analysis and reports generate automatically, the study record exists as a byproduct of running the study instead of as a document someone has to reconstruct months later.\n\nThe strategic point is about timing. Research commissioned after a dispute begins carries an obvious credibility problem; research produced in the ordinary course of building a product does not. **The cheapest form of evidentiary insurance is a routine research cadence with the methodology written down each time.** That is only realistic when a study costs days rather than months, which is the practical argument for automating the parts that do not require judgement.\n\n## Common mistakes\n\n- **Running one cell.** The single most consequential omission. See the control group section above.\n- **Defining the population after seeing the data.** Deciding who the finding is about once you know what the finding is destroys it.\n- **Treating a large sample as a substitute for a relevant one.** Size does not fix a wrong universe, and the guide is explicit that relevance to the disputed question comes first.\n- **Letting the team that wants a result administer the study.** Double-blind exists because expectation leaks through wording, ordering and tone.\n- **Never pretesting.** Twenty-five to seventy-five participants is cheap, and skipping it is treated as a weakness.\n- **Storing conclusions but not methods.** Keep instruments, screens and raw data in a [research repository](/docs/research-repository-guide), and reconcile findings across studies using [evidence synthesis](/docs/evidence-synthesis-research-findings).\n\n## Frequently asked questions\n\n### Does survey evidence get admitted in court?\n\nRoutinely, on questions of consumer perception such as confusion, deception and materiality. Rule 703 resolved the historical sampling and hearsay objections by focusing on the validity of the techniques used. Admission turns on methodology rather than on surveys as a category, and courts more often treat methodological flaws as going to weight rather than excluding the survey outright, though serious defects do get evidence thrown out.\n\n### What changed in Rule 702 in 2023?\n\nThe amendment, effective 1 December 2023, made explicit that the proponent must show it is more likely than not that each admissibility requirement is met, and clarified that the expert's opinion must reflect a reliable application of the methods to the facts of the case. The advisory committee framed it as correcting frequent misapplication rather than changing the underlying law, but in practice it has sharpened how closely courts examine expert methodology.\n\n### What response rate do I need?\n\nThere is no threshold, and the Reference Guide deliberately declines to set one. It states that representativeness matters much more for accurate inference than response rate, and that the real question is whether non-response is associated with systematic differences. Report the rate, then report what you know about how respondents differ from non-respondents.\n\n### Do I really need a control group for internal product research?\n\nYou need one for any causal claim, which includes almost every interesting claim: this design confused people, this wording changed perception, this change caused the lift. Without a control you cannot rule out that your own question produced the result. For purely descriptive work, such as how customers describe a workflow in their own words, a single cell is fine.\n\n### Can AI-moderated interviews be used as evidence?\n\nThe mode of administration is one factor among many; the guide discusses the benefits and limitations of different data collection modes rather than ruling any out. What matters is that the population, sampling, instrument, administration and analysis are documented and defensible. An AI-moderated study that is double-blind, controlled and fully recorded is in a stronger position than a poorly documented human-moderated one.\n\n### How does this connect to compliance research?\n\nDirectly. Notice and assent studies, deceptive design studies and age assurance studies all produce findings that may later be examined by a regulator or an opposing party. Running them to this standard from the start costs very little extra and is covered in [clickwrap vs browsewrap](/docs/clickwrap-vs-browsewrap-assent-research), [dark patterns testing](/docs/dark-patterns-user-testing) and [age assurance research](/docs/age-assurance-user-research).\n\nFor the specific instruments courts recognise, see [likelihood of confusion surveys](/docs/likelihood-of-confusion-survey), [secondary meaning surveys](/docs/secondary-meaning-survey) and [genericness surveys](/docs/genericness-survey-teflon-thermos); the sampling error that voids all three is covered in [survey universe](/docs/survey-universe-definition).\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types and when each is appropriate\n- [Clickwrap vs Browsewrap](/docs/clickwrap-vs-browsewrap-assent-research) - notice and assent studies built to this standard\n- [Dark Patterns Testing](/docs/dark-patterns-user-testing) - where the control cell does the decisive work\n- [Age Assurance Research](/docs/age-assurance-user-research) - testing a check a regulator will examine\n- [Sampling Bias in Research](/docs/sampling-bias-research) - why representativeness beats response rate\n- [Evidence Synthesis](/docs/evidence-synthesis-research-findings) - combining studies without overstating the record","category":"Research Operations","lastModified":"2026-08-16T03:22:58.975163+00:00","metaTitle":"Survey Evidence in Court: Daubert, FRE 702 and Defensible Research","metaDescription":"The standard courts apply to surveys is a free quality bar for product research. Learn the five-element record, the attacks each design choice defeats, and why the control group is a legal instrument.","keywords":["survey evidence in court","daubert survey","federal rule of evidence 702","litigation survey research","defensible research methodology","reference manual on scientific evidence","control group design"],"aiSummary":"The bar survey evidence must clear in federal court is competent research with the methodology documented, which makes it a useful internal quality standard. This guide covers FRE 703 and FRE 702 as amended effective 1 December 2023, the Reference Manual on Scientific Evidence Fourth Edition (2025) and its Reference Guide on Survey Research, the five-element reporting record, and a table mapping each common attack to the design choice that defeats it. Key findings: representativeness matters more than response rate, an explicit do-not-know option shifts selection by 20 to 25 percent, agree or disagree formats carry a 10 percent acquiescence inflation, and pretests run 25 to 75 respondents.","aiPrerequisites":["Familiarity with survey design","Basic understanding of sampling"],"aiLearningOutcomes":["Document the five reporting elements while a study runs","Use a control cell to defeat leading-question and guessing objections","Interpret response rate correctly relative to representativeness","Avoid acquiescence-prone agree or disagree formats","Apply double-blind administration to reduce sponsor bias"],"aiDifficulty":"advanced","aiEstimatedTime":"14 min"}],"pagination":{"total":1,"returned":1,"offset":0}}