{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-07-14T00:51:31.770Z"},"content":[{"type":"documentation","id":"36e92696-7368-4f65-98e0-b1152c36d8db","slug":"novelty-effect","title":"The Novelty Effect: Why New Features Fool Your Metrics","url":"https://www.koji.so/docs/novelty-effect","summary":"The novelty effect is a temporary rise in engagement or performance caused by the newness of a feature rather than its durable value; as curiosity fades, behavior regresses to baseline. It threatens the external validity of A/B tests and launch metrics (Kohavi, Tang & Xu 2020) and is distinct from the Hawthorne (observation) effect and its mirror image, change aversion / primacy. Detect it by running tests past the decay window (often 2-4 weeks), segmenting new vs existing users, plotting decay, and using cohort analysis; explain it with longitudinal and diary-style interviews. The honest measure of durable value is a retention curve that flattens into a plateau, not the week-one peak — a distinction that matters acutely for AI features in 2026.","content":"# The Novelty Effect: Why New Features Fool Your Metrics\n\n**The novelty effect is a temporary change in behavior, engagement, or performance caused by the *newness* of a feature or product rather than by its durable value.** When something is new, people engage more out of curiosity and the simple stimulation of change; as the novelty wears off, behavior slides back toward its real baseline. It is the reason a launch can look like a triumph in week one and a disappointment by month two — and the reason week-one metrics are among the most misleading numbers in product research.\n\nThis guide explains what the novelty effect is, how much it distorts experiments and launches, how to detect it, and how to tell a curiosity spike from genuine, durable value.\n\n## What the novelty effect is (and is not)\n\nThe critical word is *newness*. In educational technology, where the concept was formalized, the novelty effect is \"the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement... but in response to increased interest in the new technology.\" The apparent gain is real in the data; its *cause* is curiosity, not value.\n\nIts mirror image is **change aversion**, known in the experimentation literature as the **primacy effect**: when established users hit a change, their first reaction is often *negative* because they are habituated to the old way, and engagement only recovers as they relearn. Microsoft researchers, in a 2021 paper on long-term experiment estimators, define novelty as \"the desire to use new technology that tends to diminish over time\" and primacy as \"the growing engagement with technology as a result of adoption of the innovation.\" Both are time-varying biases: the effect you measure in week one is not the effect you will live with in month three.\n\n**Do not confuse it with the Hawthorne effect.** Both are reactivity biases, but the trigger differs. The Hawthorne effect is a behavior change because people know they are being *observed*. The novelty effect is a behavior change because the *thing itself is new*, independent of any observer. A user who explores a new feature more, alone at home with nobody watching, is showing novelty, not Hawthorne. They often co-occur — which is one reason short pilots overstate impact — but the fixes differ.\n\n**Historical root.** The idea is usually traced to Richard E. Clark's 1983 paper \"Reconsidering Research on Learning from Media,\" which argued that apparent learning gains from new media were confounded by novelty and instructional method. Clark's famous line: media are \"mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition.\"\n\n## Why it threatens the validity of your data\n\nThe novelty effect is a classic threat to **external validity** — the degree to which a result generalizes beyond the moment you measured it. Kohavi, Tang and Xu treat novelty and primacy as two of the principal external-validity threats in *Trustworthy Online Controlled Experiments* (2020), the standard reference distilled from companies each running tens of thousands of experiments a year.\n\nThe damage takes three forms:\n\n1. **It inflates A/B-test winners.** A variant that wins in week one may be winning on curiosity. Ship it, the novelty fades, and the metric regresses to baseline — you \"won\" a test whose effect evaporated.\n2. **It contaminates launch metrics twice over.** Early numbers are shaped by **early adopters** (systematically more curious and tolerant) *plus* the **novelty spike** on top. Neither reflects how your mainstream user base will behave once the feature is ordinary.\n3. **It fakes durable value in early research.** A burst of enthusiasm in a first session is easily mistaken for a product people will keep using. Curiosity guarantees a first click; only value earns the tenth.\n\n## The evidence: how novelty decays\n\n- **Gamification's effect follows a U-shaped curve.** In a 14-week study of 756 STEM students, Rodrigues et al. (2022) found gamification's benefit began to *decrease after about four weeks* as novelty wore off, dipped for roughly two to six weeks, then partially recovered between weeks six and ten as a \"familiarization effect\" set in. Novelty and durable value produced different curves.\n- **In education, novelty was strongest in the first six months.** One longitudinal study found study time increased after new mobile learning devices were introduced, but the effect \"was most prominent during the first six months,\" after which motivation declined — attributed to novelty.\n- **Shorter studies show larger effects.** Meta-analyses in educational technology repeatedly find that short-duration studies produce *larger* effect sizes than longer ones, with novelty in the treatment group a leading explanation. A recent meta-analysis of generative-AI learning tools similarly found initial improvements that \"gradually diminished over time, possibly due to a decrease in novelty.\"\n- **The practical decay window is short.** Experimentation practitioners commonly place the point at which returning visitors habituate to a change at roughly one to three weeks, and recommend running tests at least two to four full business weeks so novelty washes out. (These are field rules of thumb, not documented constants — there is no universal \"novelty lasts exactly N days\" figure, so treat them as guidance, not law.)\n\n## How to detect and control the novelty effect\n\n- **Run experiments long enough for the effect to wash out.** The single most important control: keep the test running past the novelty window (commonly at least two to four weeks, longer for products people use infrequently) so returning users cycle through repeated exposures.\n- **Segment new versus existing users.** Novelty chiefly hits *existing* users encountering a change; brand-new users have no prior baseline to be surprised by. If a variant shows a strong-but-decaying effect among returning users and a stable effect among new ones, novelty is inflating your aggregate — and the new-user number is the cleaner estimate of durable impact.\n- **Plot the metric over time and look for decay.** A genuine improvement holds roughly flat day over day; a novelty-driven \"win\" trends downward toward baseline. Watch the daily treatment effect, not just the pooled average.\n- **Use long-term holdouts.** Keep a slice of users on the old experience for weeks or months to measure the stabilized long-term effect — expensive, but the most reliable check on whether an early win survives.\n- **Run cohort analysis.** Comparing [cohorts](/docs/cohort-analysis-guide) by exposure date separates \"how users behave in week one\" from \"how they behave after habituation.\"\n- **Add the qualitative layer.** Decay curves tell you *that* engagement fell; they never tell you *why*. Repeat interviews and [diary studies](/docs/diary-study-guide) — a longitudinal method Nielsen Norman Group recommends for \"long-term experiences and repetitive activities\" — let you distinguish \"I used it because it was new and interesting\" from \"it became part of how I work.\"\n\n**Why this matters acutely for AI features in 2026.** New AI features tend to produce an unusually large initial curiosity spike followed by a steep drop-off, as users try the shiny thing once and then decide whether it earns a place in their routine. For AI-feature teams, week-one adoption is an especially poor proxy for durable value — the gap between \"everyone tried it\" and \"people rely on it\" has rarely been wider.\n\n## Novelty versus durable value: read the retention curve\n\nWeek-one engagement bundles three things you need to keep separate: durable value, early-adopter selection, and novelty. The cleanest read on durable value is the **retention curve** — the share of a cohort still active over time. A novelty-driven feature shows a curve that keeps sliding toward zero: everyone tried it, few stayed. A genuinely valuable feature shows a retention curve that *flattens* into a stable plateau, meaning a durable core has folded it into their routine. The height and, above all, the *flatness* of that plateau — not the week-one peak — is the honest measure of whether newness converted into habit. This is exactly the \"familiarization\" that let gamification recover after its novelty dip in the Rodrigues study: a spike and a plateau are the difference between novelty and value. (For finding that durable-value moment, see [aha-moment research](/docs/aha-moment-research) and [feature adoption research](/docs/feature-adoption-research).)\n\n## The modern approach: longitudinal AI interviews\n\nDetecting novelty quantitatively is well understood; explaining it is where teams get stuck. You can see engagement decaying in your analytics, but the dashboard cannot tell you whether users left because the feature was only ever a curiosity or because a fixable friction pushed them away. Answering that has traditionally required longitudinal interviews or diary studies — slow, expensive, and rarely run at launch cadence.\n\nKoji makes the qualitative half practical. Because interviews are AI-moderated and self-serve, you can talk to the *same* users at week one and again at week four or eight — a true [longitudinal](/docs/longitudinal-research-guide) read — and ask directly whether a feature has become part of how they work or was just fun to try once. Koji's [structured questions](/docs/structured-questions-guide) (all six types: open_ended, scale, single_choice, multiple_choice, ranking, and yes_no) capture the rating *and* the reasoning, and the AI probes the drop-off: \"You used this a lot at first and less now — what changed?\" Pairing that with your retention curves and [A/B tests](/docs/ab-testing-vs-user-research) lets you separate a novelty spike from real value in days, not quarters — so you double down on features that earn the plateau, not the ones that only won the first week.\n\n## Related Resources\n\n- [A/B Testing vs. User Research](/docs/ab-testing-vs-user-research) — when experiments mislead and interviews explain\n- [Longitudinal Research Guide](/docs/longitudinal-research-guide) — studying behavior over time\n- [Diary Study Guide](/docs/diary-study-guide) — the qualitative method for repeated, long-term use\n- [Cohort Analysis Guide](/docs/cohort-analysis-guide) — reading retention and separating exposure periods\n- [Feature Adoption Research](/docs/feature-adoption-research) — why users do (and don't) keep using features\n- [Product Analytics vs. User Research](/docs/product-analytics-vs-user-research) — pairing the \"what\" with the \"why\"\n- [Structured Questions Guide](/docs/structured-questions-guide) — the six Koji question types and when to use each\n","category":"Research Methods","lastModified":"2026-07-13T03:26:22.355949+00:00","metaTitle":"The Novelty Effect: Why New Features Fool Your Metrics","metaDescription":"The novelty effect makes new features spike in engagement out of curiosity, not durable value, misleading A/B tests and launch metrics. Learn to detect it and tell novelty from real value.","keywords":["novelty effect","change aversion","primacy effect","ab testing bias","feature launch metrics","experiment validity","retention curve"],"aiSummary":"The novelty effect is a temporary rise in engagement or performance caused by the newness of a feature rather than its durable value; as curiosity fades, behavior regresses to baseline. It threatens the external validity of A/B tests and launch metrics (Kohavi, Tang & Xu 2020) and is distinct from the Hawthorne (observation) effect and its mirror image, change aversion / primacy. Detect it by running tests past the decay window (often 2-4 weeks), segmenting new vs existing users, plotting decay, and using cohort analysis; explain it with longitudinal and diary-style interviews. The honest measure of durable value is a retention curve that flattens into a plateau, not the week-one peak — a distinction that matters acutely for AI features in 2026.","aiPrerequisites":["Basic understanding of A/B testing and product metrics","Familiarity with retention and engagement analysis"],"aiLearningOutcomes":["Define the novelty effect and distinguish it from change aversion and the Hawthorne effect","Explain how novelty threatens the validity of experiments and launch metrics","Detect novelty by running tests long enough, segmenting users, and plotting decay","Use retention curves and longitudinal interviews to separate novelty from durable value"],"aiDifficulty":"intermediate","aiEstimatedTime":"10 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}