Common Cause vs Special Cause: When a Move in Your Research Metric Is Real

Short answer: most of the movement in your research metrics is noise, and reacting to it makes the metric worse. A stable process produces variation all by itself - common cause variation, which has no single explanation and no fix short of changing the process. Occasionally something genuinely different happens - special cause variation, which does have an explanation and is worth chasing. A process behaviour chart with three-sigma limits tells you which one you are looking at, and it takes about twenty minutes to build. Without it, teams systematically mistake the first for the second, investigate causes that do not exist, and adjust in response - a behaviour called tampering, which provably increases the variation it is trying to remove.

This is the counterpart to measurement system analysis, and it points the opposite way. That guide is about reducing measurement variation. This one is about the variation you must leave alone.

The inversion that catches everyone

The instinct of a competent product team is that variation is a problem to be solved. Satisfaction dropped two points this month, so find out why. Completion rate jumped, so work out what we did right and do more of it.

Applied to common cause variation, that instinct is not merely wasted effort. It is actively harmful, and the harm is measurable.

W. Edwards Deming demonstrated this with an experiment involving a funnel, a marble and a target. The funnel is held over the target, the marble is dropped, and it lands somewhere near but not exactly on the target. The question is what to do next.

Rule 2 is the interesting one, because Rule 2 is what reasonable, diligent, well-intentioned people do. The arithmetic behind the penalty is elementary: the error in the marble drop is independent from one drop to the next, so repositioning the funnel based on the last drop adds the previous error to the next one. The standard deviation of a sum of independent variables is the square root of the number of them times the individual standard deviation, so two independent errors combine to 1.41 times one of them.

Rules 2, 3 and 4 are all tampering: taking action because of the most recent result. Tampering invariably increases the variation of a stable process.

The research versions of the four rules are easy to recognise:

• Rule 2 in the wild: satisfaction dipped, so the team rewrites two questions before the next wave. Next wave moves again, so they rewrite two more.
• Rule 3 in the wild: two teams reacting to each other, each adjusting a shared metric definition in response to the other, with the definition swinging further from anything meaningful each cycle.
• Rule 4 in the wild: setting next quarter's target to whatever this quarter happened to produce. This is the most common one, it is baked into a great many planning processes, and it is a random walk with a spreadsheet.

What a process behaviour chart actually is

A control chart, or process behaviour chart, plots a metric over time with three lines: a centre line at the process average, and upper and lower limits set three standard deviations away.

The three-sigma choice is worth understanding rather than accepting, because people assume it is a significance test and it is not. The NIST/SEMATECH e-Handbook of Statistical Methods explains the reasoning: if only chance causes are present and the variation is normal, the probability of a point falling above the upper three-sigma limit is 0.00135, and 0.0027 in both directions combined - so three-sigma limits are "the practical equivalent of 0.001 probability limits". The Handbook is explicit that "two out of one thousand is a purely arbitrary number. There is no reason why it could not have been set to one out a hundred or even larger. The decision would depend on the amount of risk the management of the quality control program is willing to take."

In other words, three sigma is an economic choice, not a statistical truth. It is set where it is because searching for a cause that does not exist is expensive, and at three sigma you will do that about twice in a thousand points. That framing matters for research, where the cost of a false alarm is a two-week investigation and a rewritten questionnaire.

For a monthly or weekly research metric you will usually have one value per period, not a subgroup, so you use a chart for individual values with limits computed from the two-point moving ranges. Donald Wheeler's rule of thumb, from his treatment of measurement consistency, is that the limits are trustworthy once they are based on at least 17 values.

Crucially, the chart is not only about points outside the limits. The NIST Handbook is clear on this: "if the plot looks non-random, that is, if the points exhibit some form of systematic behavior, there is still something wrong. For example, if the first 25 of 30 points fall above the center line and the last 5 fall below the center line, we would wish to know why this is so." A run of points on one side of the centre line is a signal even when every one of them sits inside the limits.

The decision rule, written down

"Do nothing" is a real, defensible, difficult answer, and having it written on a chart before the number moves is what makes it survivable in a stakeholder meeting. The chart converts "I think this is noise" from an opinion into a rule agreed in advance.

Before you chart anything: is the instrument consistent?

There is an order of operations here, and getting it wrong wastes the whole exercise.

In 1963 Churchill Eisenhart, a statistician at what is now NIST, wrote a line that ought to be on the wall of every research team: "Until a measurement process has been debugged to the extent that it has attained a state of statistical control it cannot be regarded, in any logical sense, as measuring anything at all."

The point is that a control chart on your metric assumes your instrument is stable. If the questionnaire changed, the panel provider changed, the sampling frame changed or the AI model behind the interviews changed mid-series, then the chart is showing you a mixture of process and instrument, and every signal is ambiguous. The fix is a consistency chart on the measurement process itself - repeated measurements of the same thing, plotted on the same kind of chart - before you chart the thing you care about. For the AI-specific version of this failure, see model version drift.

Where the boundaries with other techniques sit

This is a monitoring technique for an ongoing, unmanipulated metric. It is not a replacement for the tools you use elsewhere, and confusing them produces bad statistics:

• If you are running an experiment, you want interim analysis and stopping rules, not a control chart. Experiments have a pre-registered comparison and a defined stopping point; control charts have neither.
• If you are testing whether one number differs from another, you want statistical significance. A control chart does not test a hypothesis - it classifies a stream.
• If you are running a tracker, the chart sits on top of it. Brand tracking studies tell you how to build the series; this tells you when to react to it.
• If your extreme group bounced back, check regression to the mean before declaring a special cause. Selecting a segment because it scored badly and then observing improvement is the single most reliable way to manufacture a fake success.

Building one this week

1. Pull at least 17 consecutive periods of one metric you currently argue about. Weekly or monthly, whichever your cadence is.
2. Compute the moving ranges - the absolute difference between each consecutive pair.
3. Set the centre line at the mean and the limits at the mean plus and minus 2.66 times the average moving range. (This is the standard constant for an individuals chart; it is the three-sigma equivalent when sigma is estimated from two-point ranges.)
4. Plot every point, including the ones already discussed. Teams are often startled to find that the crisis three quarters ago was inside the limits.
5. Mark the signals using the four rows in the table above.
6. Write the decision rule into your reporting template so the next move is classified before anyone has an opinion about it.
7. Recompute limits only when a genuine, known process change occurs - a new questionnaire, a new panel, a new interview model. Never recompute them because the recent points look inconvenient.

Step 7 is the one that gets violated. Limits that get recalculated whenever the data disagrees with them are not limits.

How Koji makes this practical

A process behaviour chart needs a metric collected the same way, period after period, with a stable instrument. That is a much harder condition to meet than it sounds, and it is where most research series quietly fail.

• The instrument stays fixed. Koji structured questions carry stable IDs from the interview plan through the AI interviewer to the report, so wave 12 is measuring the same item as wave 1. All six types - open_ended, scale, single_choice, multiple_choice, ranking and yes_no - keep their identity across studies, which is what makes a longitudinal series legitimate rather than a sequence of loosely related surveys.
• The operator does not drift. In traditional tracking the moderator, the fieldwork agency and the panel composition all change over a year, and each change is an unlogged special cause. An AI interviewer that can be version-pinned removes the largest of these, and makes the remaining changes explicit decisions rather than accidents.
• A scale question gives you the series directly. Distributions come out of the report as numbers, not as a chart image you have to re-key, so building the chart is a matter of minutes rather than a data-entry project.
• The follow-up is where the special cause gets explained. This is the part traditional survey tools cannot do. When the chart flags a genuine signal, you need to know why, and a closed-ended tracker cannot tell you. Koji asks AI-generated follow-up questions inside the same interview, so the open_ended responses attached to that wave already contain the explanation - with every theme traceable back to the exact message that produced it.
• Re-running a wave is cheap. When you suspect the instrument rather than the process, you can re-measure quickly instead of waiting a quarter to find out.

The combination matters. SurveyMonkey, Typeform and Qualtrics can give you a time series; none of them will tell you whether this month's move deserves a meeting, and none of them will have already collected the reason.

Frequently asked questions

What is the difference between common cause and special cause variation?

Common cause variation is the ordinary, inherent variation a stable process produces on its own. It has no single identifiable explanation, it will occur again next period, and the only way to reduce it is to change the process itself. Special cause variation comes from something specific and identifiable that was not part of the normal process - a changed question, an outage, a pricing announcement, a different panel. The practical difference is what you should do: special causes are worth investigating, common causes are not, and treating a common cause as if it were special is what makes metrics worse.

How do I know if a change in my NPS or satisfaction score is real?

Plot at least 17 consecutive periods on a chart for individual values, set the centre line at the mean and the limits at plus and minus 2.66 times the average moving range, and then look for a point outside the limits, a long run on one side of the centre line, or a sustained trend. If none of those are present, the move is common cause variation and there is nothing to explain. Do this before the number moves, not after - a rule agreed in advance is the only kind that survives a stakeholder who dislikes the answer.

Why is three sigma the standard for control limits?

Because it is economically sensible, not because it is statistically special. Under normal variation, three-sigma limits give roughly a 0.0027 chance of a false alarm in either direction, making them what the NIST/SEMATECH Handbook calls the practical equivalent of 0.001 probability limits. The Handbook is explicit that the figure is arbitrary and could reasonably be set elsewhere depending on how much risk of a pointless investigation an organisation is willing to accept. In research, where a false alarm costs a two-week investigation and often a rewritten questionnaire, three sigma is a reasonable place to sit.

What is tampering, and how do I know if my team is doing it?

Tampering is adjusting a stable process in response to its most recent result. Deming demonstrated with the funnel experiment that this reliably increases variation: compensating for the last error relative to the current position produces a combined standard deviation 1.41 times the leave-it-alone case, and the more aggressive rules diverge entirely. You are tampering if you rewrite questions after a bad wave, change the panel because a number disappointed, or set next period's target to whatever this period produced. The test is simple: if the action was triggered by a single reading rather than by a signal on a chart, it is tampering.

Can I use a control chart instead of a significance test?

They answer different questions and are not substitutes. A significance test asks whether two specified groups differ by more than chance, within a study designed to make that comparison. A control chart asks whether a stream of values over time is behaving predictably, and classifies each new value as signal or noise. Use a control chart for an ongoing metric you monitor; use a significance test, or an experiment with pre-registered stopping rules, when you are deliberately comparing conditions.

How many data points do I need before the limits mean anything?

Wheeler's working rule is at least 17 values before you treat the limits as established, though a chart built on fewer will still show you gross signals. The more important discipline is what happens afterwards: recompute limits only when a genuine, documented process change occurs, such as a new questionnaire, a new panel, or a new interview model. Recalculating limits because recent data sits awkwardly against them destroys the entire value of the technique, since limits that move to accommodate the data can never contradict it.

Related Resources

• Structured Questions Guide - stable question IDs across the six types, which is what makes a longitudinal series comparable at all.
• Measurement System Analysis for Research Metrics - the companion technique: how much of the variation you are charting is the instrument.
• Interim Analysis and Stopping Rules - the right tool when you are running an experiment rather than monitoring a stream.
• Regression to the Mean - why extreme periods bounce back on their own, and how that manufactures fake special causes.
• Brand Tracking Studies - how to build the series that this chart sits on top of.
• Model Version Drift - the modern version of an instrument changing underneath a time series.