Automation Surprise in Research: When Your Pipeline Is Not Doing What You Think (2026)

Automation surprise is what happens when a research pipeline does something you did not expect, because it is operating in a different configuration than the one in your head. The data is not corrupted. Nobody lied. The tool did exactly what it was told. You simply were not the one who told it, or you told it something different three weeks ago and forgot. Aviation human factors has studied this failure for forty years under the name mode error, and the research literature on it is far more advanced than anything customer research has written down. This article imports it.

The distinction that matters: this is not a trust problem. An over-reliance problem is a calibration failure, where you trust the machine more or less than it deserves. A mode error is a state-tracking failure, and a perfectly calibrated operator still commits it. You can know exactly how much to trust your analysis pipeline and still read its output wrongly, because you are right about the instrument and wrong about which setting it is on.

The problem aviation named first

In 1995, Nadine Sarter and David Woods published a paper whose title is the entire phenomenon: How in the World Did We Ever Get into That Mode? Their work on glass-cockpit aircraft established the vocabulary. In a later study of the Airbus A-320, they described the shape of it plainly: one of the major problems with pilot-automation interaction is "a lack of mode awareness (i.e., the current and future status and behavior of the automation)," and as a result "pilots sometimes experience so-called automation surprises when the automation takes an unexpected action or fails to behave as anticipated."

This is not a marginal concern. A NASA Langley formal-methods analysis of mode confusion by Ricky Butler and colleagues notes that "The January 30, 1995 issue of Aviation Week lists 184 incidents and accidents involving mode awareness," and frames the underlying issue as a "lack of understanding of what the machine is doing, and why, and what it is going to do next."

Those three questions - what is it doing, why, and what will it do next - are exactly the three questions most research teams cannot answer about their own analysis pipeline.

The most important empirical finding for research operations is the one Sarter and Woods reported from their Flight Management System study. Pilots did fine in ordinary conditions. The problems were latent:

Although pilots were able to 'make the system work' in standard situations, the results reveal a variety of latent problems in pilot-FMS interaction that can affect pilot performance in nonnormal time critical situations.

Read that again with your research stack substituted for the FMS. Your pipeline works on normal studies. That is not evidence it is configured correctly. It is evidence that normal studies do not discriminate between a correct configuration and an incorrect one. The mode problem surfaces on the unusual study, under deadline, which is precisely when you have the least capacity to catch it.

The six ways a mode goes wrong

Nancy Leveson and colleagues identified six categories of design that have historically produced mode confusion. NASA's analysis summarizes them as: inputs interpreted differently in different modes, indirect mode changes, behavior that is different in different modes, operator authority limits, unintended side effects, and lack of appropriate feedback.

Every one of them has an exact analogue in a research pipeline. The mapping is not a stretch; it is close to a translation.

The last row is the dangerous one, and it generalizes. A mode error is only possible when the display is the same in both modes. If the two configurations produced visibly different output, you would notice immediately and there would be no literature about it. The entire failure depends on both readings looking plausible.

Count your own modes

Here is an exercise that takes ten minutes and tends to alarm people.

List the binary configuration switches in your own research pipeline. A typical stack has at least these seven:

1. Interview mode: structured or exploratory
2. Transcript: verbatim or cleaned
3. Theme counting: by mention or by participant
4. Denominator: invited or completed
5. Quality filter: on or off
6. Date window: rolling or fixed
7. Segment split: included or excluded

Seven binary switches produce 2^7 = 128 distinct configurations. You have consciously chosen perhaps two of them. The remaining 126 are not errors waiting to happen - they are states your pipeline can legitimately occupy, each producing a report that looks exactly as credible as the one you intended.

The combinatorics are unforgiving and they scale badly:

The aviation response to this was not to ask pilots to try harder. It was to redesign the annunciation - to make the aircraft continuously display which mode it was in, because the alternative is asking a human to hold a combinatorial state space in working memory while doing something else.

Why this is not a trust problem

It is worth being precise about the boundary, because the adjacent literature is about something else.

Research on automation bias asks whether a user trusts an AI system appropriately: over-reliance, appropriate reliance, under-reliance. That is a question about the amount of trust, and the fix is calibration - help the person trust the system exactly as much as it merits.

Mode error is orthogonal. The operator's trust can be perfectly calibrated. They know precisely how good the instrument is. They are simply mistaken about which of its legitimate behaviors it is currently exhibiting. Calibration does not help, because there is nothing miscalibrated. The two failures need different instruments: trust calibration needs a seeded-error task, mode awareness needs an annunciator.

This also means the standard remedy for automation bias - slow down, be more skeptical, check the output - does nothing for mode error. Checking the output is useless when the output is correct. It is correct for a different question.

How Koji helps

Koji is built around explicit, inspectable configuration rather than hidden defaults, which is the only real defense against this class of failure.

The interview mode is a first-class, named field on the research brief - structured, exploratory, or hybrid - not an emergent property of how you happened to word your prompts. It is visible before you launch, and it is stored with the study, so the question which mode was this study run in has an answer six months later rather than a guess.

Structured questions make the input side unambiguous. Koji supports six types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - and the type determines how the AI asks it, how the answer is extracted, and how the report visualizes it. That chain is the annunciator: a scale question cannot quietly become a free-text question between the interview and the chart, because its type travels with its stable ID from interview plan through analysis to report.

Every theme member in a Koji report carries the conversation it came from. That is what makes the state recoverable rather than merely declared. When a number looks wrong, you can open the citations behind the bar and see which interviews produced it, instead of re-running the study to find out what it was counting.

And because the aggregation layer takes an explicit population denominator rather than inferring one, the switch that silently rebases every percentage in a deck is a visible parameter instead of an accident.

Traditional tooling inverts all of this. A survey platform like SurveyMonkey and a spreadsheet full of tagged verbatims share the same flaw: the configuration lives in whoever built it, and the artifact does not carry its own settings. An AI-native platform can attach the mode to the study, which is the difference between an instrument that tells you what it is doing and one that expects you to remember.

A mode-awareness protocol

Five steps, none of which take long:

1. Write the mode inventory. List every binary switch, as above. Do it once; it changes rarely.
2. Annunciate on the artifact. Every report gets a header block stating the mode of each switch. Not an appendix - the first thing on the page.
3. Re-derive one number by hand. Pick a single headline figure per study and recompute it from the raw citations - in Koji, the citations behind any chart open directly onto the source conversation, which makes this a five-minute check rather than a re-run. This is the only check that detects a wrong-mode reading, because it is the only one that does not run through the same configuration.
4. Flag indirect changes. Any mode that can change without an operator input needs an explicit log line when it changes. This is Leveson's category two and it is the one people never instrument.
5. Diff the modes across studies before comparing them. Two studies are comparable only if their switches match. Comparing a mention-counted study to a participant-counted one is not a finding, it is an artifact.

Common mistakes

Assuming a normal-looking report is a correctly configured one. This is the Sarter and Woods finding restated. Standard situations do not discriminate between configurations.

Instrumenting the output instead of the state. Teams add validation on the numbers and none on the settings that produced them. The numbers are fine.

Treating the mode as documentation. A mode written in a Notion page is not annunciation. It has to be on the artifact that gets read, because the person reading the deck in six months is not the person who set the switch.

Confusing this with mode effects. Survey methodology already uses the word mode to mean the channel of administration - phone, web, in person. That is a real and separate phenomenon about how the channel changes the answer. This article is about machine operating state.

Adding a human reviewer and calling it solved. This is the reflex fix, and it backfires for reasons worth their own article: the reviewer's ability to catch the error decays precisely because you gave the work to the machine.

Frequently asked questions

What is automation surprise in research?

Automation surprise is when a research tool or pipeline behaves unexpectedly because it is operating in a different configuration than the researcher believes. The term comes from aviation human factors, where Sarter and Woods documented pilots being surprised by automation that took an unexpected action or failed to behave as anticipated. In research, it shows up as a report that is correctly computed under settings nobody consciously chose.

How is mode error different from automation bias?

Automation bias is a trust-calibration failure: you rely on the system more or less than it deserves. Mode error is a state-tracking failure: your trust is correctly calibrated, but you are wrong about which setting the system is on. The remedies differ. Calibration problems need evidence about the system's accuracy; mode problems need a visible indicator of current state.

Is this the same as mode effects in survey research?

No, and the shared word causes real confusion. Mode effects in survey methodology refer to the channel of data collection - telephone versus web versus face to face - and how that channel changes responses. Mode error refers to the operating state of an automated system. Both are real; they are unrelated phenomena that happen to share a term.

How many configurations does a typical research pipeline have?

More than teams expect. Seven independent binary switches - interview mode, transcript handling, theme counting unit, denominator, quality filter, date window, and segment split - produce 128 distinct configurations. Most teams have deliberately chosen two or three of those settings and inherited the rest as defaults.

Why does checking the output not catch mode errors?

Because the output is not wrong. In a mode error the system computes correctly; it computes the answer to a different question than the one you asked. Reviewing the numbers for plausibility will pass them, because they are plausible and in fact correct. Only re-deriving a figure independently, or reading the configuration directly, detects it.

What is the single most effective fix?

Annunciation: printing the active configuration on the artifact itself, where it will be read. Aviation did not solve mode confusion by training pilots to concentrate harder; it solved it by redesigning displays to continuously show the current mode. The research equivalent is a settings header on every report.

Related Resources

• Structured Questions in AI Interviews - the six question types and how type travels from plan to report
• Structured, Exploratory, and Hybrid: Choosing the Right Interview Mode in Koji - how to pick a mode, once you know which one you are in
• Mode Effects: When Letting People Choose Voice or Text Changes the Answer - the other meaning of mode, and why it matters separately
• AI Over-Reliance and Automation Bias - the trust-calibration problem this article is not about
• Model Version Drift: What Happens to Your Research When the AI Changes Mid-Study - an indirect mode change you did not authorize
• How to Analyze Interview Results - the pipeline whose switches you are inventorying