Every Report Was Safe and the Set Was Not: The Privacy Budget in Research Reporting (2026)

TL;DR: Every disclosure control in research reporting - minimum base sizes, cell suppression, quote review - is applied to one artefact at a time. Privacy is not a property of one artefact. The US Census Bureau proved this at scale: using 34 of the 180 published table sets from the 2010 Census, its own researchers reconstructed the confidential person records and confirmed that "all records in 70% of all census blocks (97 million people) are perfectly reconstructed." Every one of those tables had passed disclosure review individually. The formal statement of the problem is the composition theorem of differential privacy - in Dwork and Roth's phrasing, "the epsilons and the deltas add up" - and its practical consequence is that reviewing reports one by one is structurally incapable of establishing that your reporting is safe. The unit of account is the release history, and almost nobody keeps one.

The claim that was wrong for fifty years

Start with the assumption, because it is probably yours.

Aggregate statistics are safer than the records they came from. You cannot see a person in an average. A table of counts is a summary, and summaries lose information; that is what makes them summaries. So publish tables freely and guard the microdata.

The Census Bureau's technical report on the 2010 reconstruction attack opens by naming exactly this belief: "For the last half-century, it has been a common and accepted practice for statistical agencies, including the United States Census Bureau, to adopt different strategies to protect the confidentiality of aggregate tabular data products from those used to protect the individual records contained in publicly released microdata products. This strategy was premised on the assumption that the aggregation used to generate tabular data products made the resulting statistics inherently less disclosive than the microdata from which they were tabulated."

And then it demolishes it: "This paper demonstrates that, in the context of disclosure limitation for the 2010 Census, the assumption that tabular data are inherently less disclosive than their underlying microdata is fundamentally flawed."

The mechanism is arithmetic, not cryptography. Each published count is a linear equation over the unknown individual records. Publish enough equations and the system becomes solvable. The 2010 Census "published more than 150 billion aggregate statistics in 180 table sets." Most of those tables were published at the level of the individual census block - units that, as the report notes, "can have populations as small as one person." Using only 34 of those table sets, and five variables - census block, sex, age, race, ethnicity - the team reconstructed the underlying microdata and could verify from published data alone that 70% of blocks, 97 million people, were perfectly reconstructed. Linking those reconstructed records to commercial data then let them "correctly infer the actual census response on race and ethnicity for 3.4 million vulnerable population uniques" with 95% accuracy.

No single table disclosed anything. That is the point. The disclosure had no locus.

The new failure mode: every release was safe and the set was not

This is a distinct kind of measurement failure, and it deserves its own name because it defeats the review process rather than any particular control.

The failures research teams are trained to catch are all local. A cell with n = 2 is visibly wrong. A quote naming an employer is visibly wrong. A 6-of-6 unanimous segment is visibly wrong once you know to look. You can find all of them by examining the artefact in front of you.

Composition failures are not local. Each artefact is correct. The property you care about - can a reader isolate an individual? - is a function of the set of artefacts, and no examination of a member of a set can tell you a property of the set. This is why we review every report before it goes out is not a privacy programme. It is a necessary control that is structurally blind to the failure mode that actually got the Census Bureau.

The prescription follows from the diagnosis. If the property belongs to the set, you have to hold the set. That means a release log: what was published, over which respondent population, cut by which attributes, to which audience, when. Not a folder of reports - an index of the aggregates in them. Almost no research team has one, which means almost no research team can answer the only question that matters.

What differential privacy actually contributes here

Differential privacy is usually introduced as a noise-injection technique, which makes it sound like a tool you either adopt wholesale or ignore. Its more useful contribution to ordinary research reporting is conceptual: it is the framework that made privacy loss additive and therefore trackable.

The composition theorem, in Dwork and Roth's monograph, says that combining an algorithm with privacy parameter epsilon-one and another with epsilon-two yields a combined guarantee of epsilon-one plus epsilon-two, and generalises to any number of releases - or, in their summary phrase, "the epsilons and the deltas add up." They also note the property that makes it practically important - composition, they write, is automatic, "in that the bounds obtained hold without any special effort by the database curator."

Automatic is the operative word, and it cuts both ways. You do not have to do anything to make composition happen. It happens whether or not you are tracking it.

The epsilon parameter bounds how much more likely any output becomes when one person's data is included versus excluded - a multiplicative factor of e to the power epsilon. Watch what happens when a "safe" release is repeated:

Eight quarterly dashboards, each individually defensible, leave a guarantee 33 times weaker than the first one. Four years of quarterly reporting leaves a guarantee that is not a guarantee.

You do not need to implement differential privacy to use this. The insight transfers directly: your privacy posture degrades monotonically with every release, and nothing in your process currently notices. A team that publishes a segment tracker every quarter for four years has spent something. It has no idea what.

The three questions a release log answers

A release log is a boring artefact that does three things no report review can do.

1. Which respondents have been reported on most? Privacy loss concentrates on the people who appear in the most cuts - usually your most engaged customers, because they participate in everything. The panel member who has been in eleven studies is exposed eleven times over, and nobody has ever looked at that number. Sort your log by respondent and the exposure distribution is usually startling.

2. Which pairs of releases are differenceable? Two releases over respondent sets differing by a handful of people are a differencing attack waiting for someone to notice. You cannot see this from either release. You can see it instantly from a log that records the population of each.

3. How fine has the grid become? Segment definitions proliferate. Each new cut is defensible on its own and the cumulative effect is a much finer partition of the same population than anyone approved. A log makes the drift visible.

None of these requires sophistication. A table with one row per published aggregate - date, study, population definition, segment attributes, minimum cell size, audience - is enough to answer all three.

Practical controls that respect composition

Freeze the segment grid. One standing set of coarse segments used by every report is the single highest-leverage control, because it caps the number of distinct equations you ever publish. Ad-hoc cuts are the thing that made the Census tables solvable.

Set a budget in releases, not in reports. Decide in advance how many distinct aggregate cuts a given respondent population will support per year, and treat additional cuts as spending against it. The number will feel arbitrary. It is still infinitely better than the current implicit budget, which is unbounded.

Rotate the population. Composition bites hardest when the same people are reported on repeatedly. Refreshing panel membership does more for privacy than any amount of per-report review, and it improves data quality at the same time by reducing panel conditioning.

Prefer bands to point estimates for standing metrics. A tracker reporting "between 4 and 5" publishes far fewer usable equations than one reporting 4.23, and, as noted in the differencing article, the extra precision was mostly noise anyway.

Review the log, not just the report. Add one recurring agenda item: what have we published about this population this year? That question cannot be answered by looking at the artefact in front of you, which is exactly why it needs to be asked separately.

Where Koji fits

Koji is an AI-native research platform - AI-moderated voice and text interviews, automatic follow-up probing, generated reports - and the honest framing is that no platform can solve composition for you, because composition is a property of your publishing behaviour, not of the tool. What a platform can do is make the release history knowable.

Studies are bounded objects with enumerable outputs. A Koji study has a defined respondent set, a defined question set and a generated report whose aggregates derive from those questions. That is the raw material of a release log arriving as a by-product of normal use, rather than as a documentation chore. Compare that with a spreadsheet-and-BI workflow, where the set of aggregates that has ever been shown to anyone is genuinely unknowable.

The structured question schema caps the grid. Koji's six question types - open_ended, scale, single_choice, multiple_choice, ranking and yes_no - carry stable IDs from the interview plan through analysis into report aggregation. Because segments derive from declared single_choice and multiple_choice options rather than from whatever a filter builder permits, the number of distinct cuts you can publish is bounded by design. Bounded is the whole game: the Census attack worked because 150 billion statistics is a very large number of equations.

Depth substitutes for granularity. The structural driver of grid proliferation is that teams keep slicing because they are looking for the explanation. Koji's AI interviewer probes with up to three follow-ups per question based on what the respondent actually said, which puts the explanation in the transcript rather than in an ever-finer crosstab. A team that gets its why from open_ended responses does not need the twelfth cut of the scale question - and every cut it does not publish is an equation an attacker does not get. That is a real privacy dividend from an AI-native method, and it is not available to a survey tool whose only depth control is more questions.

One respondent population, many studies, one place to look. Because studies live in one workspace, the which respondents have been reported on most question is answerable in principle. Most research organisations cannot answer it in principle, because the data is in six tools.

The uncomfortable conclusion

The Census Bureau had better disclosure controls than your research team. It had statisticians, a legal mandate, decades of methodology, and rules applied to every table it published. Every one of those 150 billion statistics passed review.

It was not enough, because the review was per-table and the vulnerability was per-corpus. That is not a criticism of the reviewers; it is a statement about what review can and cannot establish. A property of a set is not visible from its members.

So the practical takeaway is small and specific: start a release log. It will not be rigorous, it will not have an epsilon in it, and it will still be the only artefact in your research operation capable of answering the question is what we publish about our customers safe? Everything else you have answers a strictly easier question about one report.

Frequently asked questions

Can individual people be identified from aggregate research reports?

Yes, if there are enough of them. Each published count or mean is effectively an equation constraining the underlying records, and a large enough set of equations becomes solvable. The Census Bureau demonstrated this on its own 2010 data, reconstructing person records from published tables and verifying that 97 million people - 70% of census blocks - were perfectly reconstructed. Aggregation is not anonymisation.

What is composition in privacy terms?

The principle that privacy loss accumulates across releases. Dwork and Roth's composition theorem states that combining releases with parameters epsilon-one and epsilon-two yields a guarantee of epsilon-one plus epsilon-two - "the epsilons and the deltas add up" - and the bounds hold "without any special effort by the database curator." Eight releases at epsilon = 0.5 leave a bound 33 times weaker than one.

Do I need differential privacy to run customer research responsibly?

No. Formal differential privacy is heavy machinery aimed at high-volume public statistical products. What transfers to ordinary research reporting is the accounting insight: privacy loss is cumulative, so the unit of review must be the release history rather than the individual report. You can act on that with a spreadsheet.

What is a release log and what goes in it?

A table with one row per published aggregate: date, study, the respondent population it covers, the segment attributes it was cut by, the minimum cell size, and the audience. That is enough to answer which respondents have been reported on most often, which pairs of releases can be differenced against each other, and how fine the published segment grid has become over time.

Is per-report review useless then?

Not at all - it catches the local failures, which are the common ones: cells below the base-size threshold, unanimous small segments, identifying verbatims. It is simply blind to failures that are properties of the set of reports rather than of any one report. You need both, and most teams have only the first.

How many reports is too many on one panel?

There is no derivable answer, because it depends on how fine the cuts are and how stable the population is. The useful move is to make the number explicit: decide in advance how many distinct aggregate cuts a given respondent population supports per year and treat further cuts as spending against that allowance. An arbitrary budget that is tracked beats an unlimited one that is not.

Related Resources

• The Differencing Attack - the two-release special case of composition, and why suppressing a small cell publishes it.
• k-Anonymity for Segment Reporting - the per-release base-size rule that this article shows is necessary but not sufficient.
• Quasi-Identifiers in Research Data - why the combination of screener fields is the identifier, and how to measure uniqueness.
• Structured Questions in AI Interviews - the six question types, and how a declared schema bounds the number of publishable cuts.
• Panel Conditioning and Repeat Participants - the data-quality case for rotating the population, which is also the privacy case.
• Generating Research Reports - how Koji turns a completed study into an analysed report.