{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-01T09:57:16.712Z"},"content":[{"type":"documentation","id":"e7cb389b-0f85-4b80-ad3c-936d46d2a0ed","slug":"case-definition-feedback-investigation","title":"The Case Definition: The Decision That Fixes or Breaks Every Issue Count (2026)","url":"https://www.koji.so/docs/case-definition-feedback-investigation","summary":"A case definition is the written set of criteria deciding what counts as an instance of an issue. Field epidemiology (CDC Principles of Epidemiology) stages it deliberately: a sensitive or loose definition including confirmed, probable and possible cases during hypothesis generation, then a specific definition dropping possible and sometimes probable for analytic work, because inclusion of false-positive cases can produce misleading results. The critical rule is that a case definition must not include the exposure or risk factor being evaluated, which is why filtering a corpus by the feature you suspect makes the finding circular. A worked example shows 1,000 tickets yielding either 420 or 48 affected accounts (8.75x) depending on tier inclusion, and shows how switching definitions silently between reports manufactures an 89 percent improvement.","content":"Every number you report about an issue - how many users hit it, whether it is getting better, which segment suffers most - is downstream of one decision you probably made in a hurry: what counts as a case. Field epidemiology treats that decision as a formal artifact called a case definition, writes it down, and changes it deliberately at known points in an investigation. Product teams usually leave it implicit, which is why two analysts can report an 8x difference in affected users from the same ticket queue and both be right.\n\n## The short answer\n\nWrite the definition down before you count. Use a deliberately broad definition while you are still working out what is happening, and a deliberately narrow one when you are testing what caused it. Classify cases into tiers rather than in or out. And never, under any circumstances, define a case by the cause you are trying to evaluate.\n\n## What a case definition actually is\n\nThe CDC's Principles of Epidemiology course defines it plainly: \"A case definition is a set of standard criteria for classifying whether a person has a particular disease, syndrome, or other health condition.\" In an outbreak investigation it is usually restricted further by time, place and person.\n\nThe product translation is direct. A case definition for an issue investigation specifies the observable symptom, the window, and the population - for example, an account that submitted a payment and received a failure response, between the 14th and the 28th, on the web client. What it must not specify is why you think it happened.\n\n## Use two definitions, not one\n\nThe single most useful thing epidemiology does here is refuse to pick one definition. The CDC describes the staging explicitly. Early on, \"investigators may use a 'loose' or sensitive case definition that includes confirmed, probable, and possible cases to characterize the extent of the problem, identify the populations affected, and develop hypotheses about possible causes.\" The course then describes the second stage just as explicitly: \"Later on, when hypotheses have come into sharper focus, the investigator may tighten the case definition by dropping the 'possible' and sometimes the 'probable' category.\"\n\nThe reason for tightening is stated just as plainly: \"In analytic epidemiology, inclusion of false-positive cases can produce misleading results.\"\n\nSo the two definitions do different jobs, and using the wrong one for the wrong job is the error:\n\n- **Sensitive, early.** You want to know whether there is a problem at all, roughly how big it is, and who is in it. Missing real cases is the expensive mistake here, so cast wide and accept some false positives.\n- **Specific, later.** You want to know what caused it, or whether a fix worked. Now a false positive is the expensive mistake, because contaminated cases wash out the association you are trying to measure.\n\nThe CDC's tiers transfer cleanly. \"To be classified as confirmed, a case usually must have laboratory verification. A case classified as probable usually has typical clinical features of the disease without laboratory confirmation. A case classified as possible usually has fewer of the typical clinical features.\"\n\n| Tier | Epidemiology | Product research |\n| --- | --- | --- |\n| Confirmed | Laboratory verification | Reproduced, or server logs show the failure for that account |\n| Probable | Typical features, no lab confirmation | Clear symptom description matching the known signature, no log evidence |\n| Possible | Fewer typical features | Vague report consistent with the issue but also with several others |\n\n## The rule almost every product team breaks\n\nHere is the sentence worth printing out. The CDC states: \"The case definition must not include the exposure or risk factor you are interested in evaluating. This is a common mistake.\"\n\nConsider what happens when you ignore it. Your team suspects the new checkout provider. Someone defines the investigation set as tickets mentioning the new checkout. Now 100 percent of your cases involve the new checkout, and the provider looks overwhelmingly implicated. But that result was guaranteed by the definition before any data was examined. You did not measure an association; you asserted one and then counted it.\n\nThe correct move is to define the case by the symptom - a failed payment submission in the window - and then measure how many of those cases touched the new provider versus the old one. Only then does the comparison carry information. If 300 of 400 symptom-defined failures ran through the new provider while it handled only 40 percent of traffic, you have a real signal. If they are proportional to traffic, you have just saved yourself a rollback.\n\nThis failure is especially common when the investigation set comes from a search query, because a keyword search *is* an exposure-based definition wearing a convenient disguise. Searching your corpus for the feature name you suspect is the same mistake with a different interface.\n\n## A worked example: one queue, eight different answers\n\nTake 1,000 support tickets from a two-week release window. After classification:\n\n- **Confirmed** - reproduced in-house or matched to a server-side failure for that account: **48**\n- **Probable** - describes the exact signature, submitted payment then saw a specific error, no log match available: **112**\n- **Possible** - says something like \"checkout is broken\" with no detail: **260**\n\nA sensitive definition counts all three: **420 affected accounts**. A specific definition counts confirmed only: **48 affected accounts**. From one queue, with no disagreement about any individual ticket, the headline number varies by a factor of 8.75. Both are defensible. Neither is wrong. They answer different questions.\n\nNow the trap. Suppose week one reports the sensitive number, 420, because you were still scoping the problem. Week two, now in causal-analysis mode, you report the specific number, 48. Nobody is lying and nobody mentions the change. The dashboard shows affected accounts falling from 420 to 48 - an 89 percent improvement - produced entirely by a definition change. A fix that did nothing gets credited, and the engineer who actually fixed something next month will be unable to show it.\n\nThe discipline that prevents this costs one line: report the definition alongside the count, every time, and when you tighten it, restate the prior period under the new definition as well. Two numbers for the transition period, always.\n\n## Changing the definition without corrupting the trend\n\nChanging a case definition mid-investigation is correct practice, not a failure. Doing it silently is the failure. A workable protocol:\n\n1. Version the definition. A short identifier in the report is enough.\n2. When the definition changes, recount at least one prior period under both the old and new version, and publish both.\n3. Keep the tier assignments, not just the total. If you only store the final count you cannot ever recount, and every historical comparison becomes permanently unavailable.\n4. State the window and population in the definition itself, so a later reader cannot accidentally compare a seven-day count to a fourteen-day one.\n\n## How Koji handles this\n\nThe reason retrofitting a case definition onto free text is painful is that the defining symptom was never asked of everyone. A support queue only contains people who wrote in, and only tells you what each of them happened to volunteer. Koji lets you move the definition upstream, to collection time.\n\nBecause a Koji study can mark a question `required`, the AI interviewer covers the defining symptom with every participant rather than only the ones who raise it unprompted. That gives you something a ticket queue structurally cannot: a denominator of people who were actually asked. Koji's structured questions make the classification itself unambiguous - a `yes_no` item for whether the participant hit the failure, a `single_choice` item for which flow they were in, a `scale` item for severity - so the case assignment does not depend on parsing a sentence someone typed while annoyed. Those are three of the six types available, alongside open_ended, multiple_choice and ranking.\n\nKoji's analysis layer maps onto the tier structure more neatly than most teams expect. Each extracted answer carries a confidence flag of high, medium or low, which is a close analogue of confirmed, probable and possible, and each one stores the indices of the transcript messages it was drawn from. So when you tighten a definition from \"all tiers\" to \"high confidence only,\" you can recount the existing corpus instantly instead of re-reading it, and you can show a reviewer exactly which participant words put each case in its tier. That is what makes the honest two-number transition cheap enough to actually do.\n\nOne more advantage worth naming: Koji's AI interviewer asks follow-up questions in the moment, so an ambiguous answer can be resolved into a confirmed or ruled-out case during the interview. In a survey or a ticket queue the ambiguity is permanent, and permanent ambiguity is what forces you to choose between a number that overcounts and a number that undercounts.\n\n## Common mistakes\n\n- **Defining the case by the suspected cause.** Including the exposure guarantees the finding. A keyword search for the feature you suspect is the same error.\n- **Reporting a count without its definition.** The number is meaningless alone, and the next analyst will unknowingly produce a different one.\n- **Tightening the definition silently.** This manufactures improvement. It is the single most common way a feedback dashboard lies.\n- **Using the strict definition during discovery.** You will conclude the problem is small because you excluded most of it, and close the investigation early.\n- **Using the loose definition for causal analysis.** False positives dilute the association until a real cause looks like noise.\n- **Collapsing to in or out.** The probable and possible tiers carry the information about how confident you are, and deleting them replaces a measurable uncertainty with an argument.\n- **Leaving the window implicit.** Half of all apparent trends in issue counts are two different window lengths compared to each other.\n\n## Frequently asked questions\n\n### What is a case definition in product research?\n\nIt is a written set of criteria deciding which users, sessions or reports count as instances of the issue you are investigating. It specifies the observable symptom, the time window and the population, and it deliberately excludes any statement about the suspected cause.\n\n### Why would I use two different definitions in one investigation?\n\nBecause the two phases have opposite error costs. While scoping, missing real cases is worse, so you want a broad definition. While testing a cause or a fix, false positives are worse, because they dilute the association you are trying to detect. One definition cannot be correct for both.\n\n### Why can the case definition not include the cause I suspect?\n\nBecause it makes the conclusion circular. If a case is defined as a report mentioning the new checkout, then every case involves the new checkout by construction, and the apparent association is an artifact of your own filter rather than a measurement.\n\n### Can I change the case definition partway through?\n\nYes, and you often should. The requirement is that you version it, and that you recount at least one prior period under both the old and the new definition so the trend stays interpretable. Silent changes create improvements that did not happen.\n\n### How do confirmed, probable and possible map onto product issues?\n\nConfirmed means you reproduced it or found server-side evidence for that specific account. Probable means the report matches the known signature but you have no independent confirmation. Possible means the report is consistent with this issue and also with several others.\n\n### Does this matter if I only have support tickets to work with?\n\nIt matters more. A ticket queue contains only people who wrote in and only the details they volunteered, so you have no denominator of people who were asked. You can still classify into tiers and publish the definition, but you should state plainly that the count is of reports, not of affected users.\n\n## Related Resources\n\n- [Why Complaint Counts Cannot Become Rates](/docs/complaint-counts-cannot-be-rates) - what you can and cannot compute once the cases are defined\n- [Support Ticket Analysis](/docs/support-ticket-research-analysis) - mining the queue where case definitions are usually left implicit\n- [Product Feedback Triage](/docs/product-feedback-triage-guide) - where tier assignment belongs in your workflow\n- [Customer Feedback Categorization](/docs/customer-feedback-categorization) - taxonomy design, and how wide a category should be allowed to get\n- [Why Your Quarterly Metric Shows a Trend That Is Not There](/docs/measurement-interval-false-trend) - the other way a reporting artifact manufactures a trend\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types, and why classifying at collection time beats retrofitting\n","category":"Research Methods","lastModified":"2026-10-01T03:51:08.867918+00:00","metaTitle":"Case Definitions for Product Issue Counts","metaDescription":"Two analysts can report an 8x difference in affected users from one queue. The fix is a written case definition, staged in two versions.","keywords":["case definition product research","how to count affected users","confirmed probable possible issue","operational definition feedback","issue count methodology","outbreak investigation product"],"aiSummary":"A case definition is the written set of criteria deciding what counts as an instance of an issue. Field epidemiology (CDC Principles of Epidemiology) stages it deliberately: a sensitive or loose definition including confirmed, probable and possible cases during hypothesis generation, then a specific definition dropping possible and sometimes probable for analytic work, because inclusion of false-positive cases can produce misleading results. The critical rule is that a case definition must not include the exposure or risk factor being evaluated, which is why filtering a corpus by the feature you suspect makes the finding circular. A worked example shows 1,000 tickets yielding either 420 or 48 affected accounts (8.75x) depending on tier inclusion, and shows how switching definitions silently between reports manufactures an 89 percent improvement.","aiDifficulty":"intermediate","aiEstimatedTime":"11 min"}],"pagination":{"total":1,"returned":1,"offset":0}}