Survey Evidence in Court: Daubert, FRE 702, and Research That Survives Cross-Examination

The bar a survey must clear to be admitted in federal court is not exotic. It is competent research with the methodology written down. That makes it unusually useful as an internal quality standard, because unlike "did the stakeholders find this convincing", it is specific, published, and adversarial by design. If you want a single question that reliably separates research that will hold up from research that will not, use this one: could this survive someone whose job is to destroy it?

Most product research would not, and the reason is almost always the same missing design choice. We will get to it.

The legal frame, briefly

Two rules do the work. Federal Rule of Evidence 703 settled the old objections to surveys, that they rest on sampling and that they are hearsay, by redirecting attention to the validity of the techniques employed; the inquiry is whether the facts or data are of a type reasonably relied upon by experts in the field. Federal Rule of Evidence 702, as amended effective 1 December 2023, then governs the expert who presents them. It now states explicitly that the proponent must demonstrate to the court that it is more likely than not that the testimony rests on sufficient facts or data, is the product of reliable principles and methods, and that the expert's opinion reflects a reliable application of those principles and methods to the facts of the case. The 2023 amendment was made because courts had been applying that burden too loosely. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), remains the source of the gatekeeping role.

The practical guidance judges actually consult is the Reference Manual on Scientific Evidence, whose Fourth Edition was published by the National Academies Press on 31 December 2025. Its Reference Guide on Survey Research, by Shari Seidman Diamond, Matthew Kugler and James N. Druckman, is the closest thing that exists to an official checklist for evaluating a survey.

The five things you must be able to describe

The Reference Guide frames adequate reporting around five elements. Treat this as the minimum record for any study you might ever need to defend.

Record these while the study runs. Reconstructing them afterwards is where most internal research quietly fails, and it is also where an opposing expert will start.

The attacks, and the design that defeats each

This is the table worth keeping. Every column-two entry is a decision made before fielding, not an argument made afterwards.

The control group is a legal instrument

If you take one thing from this guide, take this. The Reference Guide makes the point crisply in the deception context: if respondents who saw the allegedly deceptive advertisement answer differently from respondents who saw a control advertisement, the difference cannot merely be the result of a leading question, because both groups answered the same question. Background noise, guessing, and even a poorly worded item should push both cells in the same direction, so the comparison survives objections that would destroy a single-cell study.

That is why so much product research is undefendable by construction. It runs one cell, produces a percentage, and has no answer at all when someone asks whether the number would have appeared anyway. Adding a control cell roughly doubles the sample and neutralises the three most common attacks at once. It is the highest-return design decision in applied research, and it is the same instrument that makes dark patterns testing and clickwrap notice studies meaningful rather than merely suggestive.

Three findings that should change how you run ordinary studies

The Reference Guide contains specific, quantified guidance that is directly useful outside litigation.

Stop panicking about response rate. The guide states that surveys with varying response rates have produced surprisingly comparable results, and that generally the representativeness of a sample, regardless of response rate, matters much more for accurate inference than the response rate itself. What matters is whether non-response is associated with systematic differences that cannot be modelled or assessed. A high response rate from a skewed frame is worse than a low one from a good frame. This reframes a metric teams routinely over-weight, and it connects directly to sampling bias and survivorship bias.

An explicit do-not-know option moves the numbers a lot. Offering one commonly leads to a 20 to 25 percent increase in the proportion of respondents selecting it. That is not a rounding effect. Whether you offer it is a substantive design decision that should be made deliberately and reported, because a study without one has quietly converted uncertainty into opinion.

Agree or disagree formats carry a documented inflation effect of about 10 percent. The guide describes acquiescence as the tendency to endorse any assertion regardless of content, and says the format only yields reasonable estimates when control groups, control questions or counterbalanced items are added. If your satisfaction tracker is built on agree or disagree items, some of your trend is format.

Two more worth internalising: pretests are typically run with 25 to 75 respondents of the same type eligible for the main study, and courts have treated the absence of pretesting as a weakness. And double-blind administration means the instrument gives no clue to sponsorship or expected answers, down to details like not reversing the usual order of yes and no boxes on a key question.

How Koji fits

Most of this checklist is about discipline and documentation, which is exactly where general-purpose survey tools leave you on your own. A Typeform or SurveyMonkey questionnaire will happily field an agree-or-disagree battery with no control cell, no do-not-know option and no record of why the population was defined as it was. Qualtrics gives you more control but still treats the qualitative half as a separate exercise, which means the explanations that make a finding legible arrive on a different timeline, if at all.

Koji is built around the parts that make research defensible. All six structured question types, open_ended, scale, single_choice, multiple_choice, ranking and yes_no, live in one AI-moderated conversation, so the countable measures and the verbatim reasoning come from the same respondents in the same session rather than from two studies stitched together. The AI interviewer probes an ambiguous answer on the spot, which is how you distinguish a respondent who genuinely held a belief from one who was guessing, precisely the attentiveness evidence the fourth element of the record calls for. Interviews run by voice or text with no moderator to schedule, so a two-cell design with a real control is affordable rather than a luxury. And because analysis and reports generate automatically, the study record exists as a byproduct of running the study instead of as a document someone has to reconstruct months later.

The strategic point is about timing. Research commissioned after a dispute begins carries an obvious credibility problem; research produced in the ordinary course of building a product does not. The cheapest form of evidentiary insurance is a routine research cadence with the methodology written down each time. That is only realistic when a study costs days rather than months, which is the practical argument for automating the parts that do not require judgement.

Common mistakes

• Running one cell. The single most consequential omission. See the control group section above.
• Defining the population after seeing the data. Deciding who the finding is about once you know what the finding is destroys it.
• Treating a large sample as a substitute for a relevant one. Size does not fix a wrong universe, and the guide is explicit that relevance to the disputed question comes first.
• Letting the team that wants a result administer the study. Double-blind exists because expectation leaks through wording, ordering and tone.
• Never pretesting. Twenty-five to seventy-five participants is cheap, and skipping it is treated as a weakness.
• Storing conclusions but not methods. Keep instruments, screens and raw data in a research repository, and reconcile findings across studies using evidence synthesis.

Frequently asked questions

Does survey evidence get admitted in court?

Routinely, on questions of consumer perception such as confusion, deception and materiality. Rule 703 resolved the historical sampling and hearsay objections by focusing on the validity of the techniques used. Admission turns on methodology rather than on surveys as a category, and courts more often treat methodological flaws as going to weight rather than excluding the survey outright, though serious defects do get evidence thrown out.

What changed in Rule 702 in 2023?

The amendment, effective 1 December 2023, made explicit that the proponent must show it is more likely than not that each admissibility requirement is met, and clarified that the expert's opinion must reflect a reliable application of the methods to the facts of the case. The advisory committee framed it as correcting frequent misapplication rather than changing the underlying law, but in practice it has sharpened how closely courts examine expert methodology.

What response rate do I need?

There is no threshold, and the Reference Guide deliberately declines to set one. It states that representativeness matters much more for accurate inference than response rate, and that the real question is whether non-response is associated with systematic differences. Report the rate, then report what you know about how respondents differ from non-respondents.

Do I really need a control group for internal product research?

You need one for any causal claim, which includes almost every interesting claim: this design confused people, this wording changed perception, this change caused the lift. Without a control you cannot rule out that your own question produced the result. For purely descriptive work, such as how customers describe a workflow in their own words, a single cell is fine.

Can AI-moderated interviews be used as evidence?

The mode of administration is one factor among many; the guide discusses the benefits and limitations of different data collection modes rather than ruling any out. What matters is that the population, sampling, instrument, administration and analysis are documented and defensible. An AI-moderated study that is double-blind, controlled and fully recorded is in a stronger position than a poorly documented human-moderated one.

How does this connect to compliance research?

Directly. Notice and assent studies, deceptive design studies and age assurance studies all produce findings that may later be examined by a regulator or an opposing party. Running them to this standard from the start costs very little extra and is covered in clickwrap vs browsewrap, dark patterns testing and age assurance research.

For the specific instruments courts recognise, see likelihood of confusion surveys, secondary meaning surveys and genericness surveys; the sampling error that voids all three is covered in survey universe.

Related Resources

• Structured Questions Guide - the six question types and when each is appropriate
• Clickwrap vs Browsewrap - notice and assent studies built to this standard
• Dark Patterns Testing - where the control cell does the decisive work
• Age Assurance Research - testing a check a regulator will examine
• Sampling Bias in Research - why representativeness beats response rate
• Evidence Synthesis - combining studies without overstating the record