{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-22T07:46:15.585Z"},"content":[{"type":"documentation","id":"d7716ce5-f4e8-4b7d-9e7c-9cdba68ba83e","slug":"ecological-validity-user-research","title":"Ecological Validity in User Research: Will Your Findings Hold Up in Real Life?","url":"https://www.koji.so/docs/ecological-validity-user-research","summary":"Ecological validity is how closely a study's setting, tasks, materials and participants match real life. Bronfenbrenner (1977) criticised lab-bound research, citing a survey where 76% of 902 child development studies used the laboratory paradigm and 8% were observational. Sheeran (2002) found stated intentions explain about 28% of the variance in behaviour across 422 studies. Duh, Tan and Chen (2006) found lab and field mobile tests differed in the frequency and severity of usability problems, while Andreasen et al. (2007) found remote synchronous testing virtually equivalent to the lab. Mobile was about 62.5% of global web traffic in Q2 2025 (StatCounter via Statista). The guide compares internal, external and ecological validity, lists six ways studies lose realism, gives an eight-point checklist, explains when lower realism is acceptable and shows how Koji's self-paced voice and text interviews, in-product embed, call link, screening and evidence-linked findings help.","content":"Ecological validity is how well what you observe in a study matches what people actually do in their own lives. A usability test in a quiet room on a borrowed laptop, with a moderator watching, can produce clean results that fall apart the moment a real user tries the same thing on a phone, on a train, with a toddler asking for a snack. High ecological validity means your setting, tasks, timing and participants look like the real thing, so the findings are likely to survive contact with real use.\n\nYou rarely get it for free. Most research choices trade a little realism for control, speed or cost. This guide covers what the term means, how it differs from internal and external validity, what the evidence says about lab versus real-world results, where ecological validity breaks in user research and a checklist for raising it.\n\n## What ecological validity means\n\nThe phrase comes from psychology. Egon Brunswik used it in the 1940s and 1950s for a narrow, technical idea about how well cues in the environment predict what they signal. Urie Bronfenbrenner, writing in _American Psychologist_ in 1977, pushed the field towards the meaning most researchers use today: whether the conditions of a study resemble the settings people actually live in.\n\nHis criticism of laboratory child psychology is still the clearest statement of the problem. Much of the field, he wrote, was \"the science of the strange behavior of children in strange situations with strange adults for the briefest possible periods of time.\"\n\nSwap \"children\" for \"users\" and that sentence describes a lot of product research: an unfamiliar room, a scripted task, a stranger with a notepad and thirty minutes on the clock.\n\nIn user research, ecological validity has four practical parts:\n\n- **Setting.** Is the session happening where the behaviour normally happens, or somewhere convenient for the researcher?\n- **Task.** Is the participant doing something they would actually do, with their own goal, or a task you invented?\n- **Materials.** Are they using their own device, account and data, or a test build with dummy content?\n- **People.** Are they the people who really do this, or proxies who happened to be available?\n\n## Ecological, internal and external validity compared\n\nThe three terms get mixed up, and they pull in different directions.\n\n![Table comparing internal, external and ecological validity: the question each answers, what threatens it and how to raise it](/visuals/internal-external-ecological-validity-compared/internal-external-ecological-validity-compared.png \"Tightening one kind of validity often loosens another, so decide which one this study needs most\")\n\n- **Internal validity** asks whether the thing you changed caused the effect you saw. Controlled lab studies are strong here because you remove distractions and hold everything else constant.\n- **External validity** asks whether the result generalises to other people, places and times. It is mostly a question of sampling.\n- **Ecological validity** asks whether the conditions of the study resemble real life. It is one route to external validity, but not the only one, and a study can be realistic and still not generalise if you only spoke to five power users.\n\nThe tension is real. Every distraction you remove to tighten internal validity makes the session less like the participant's actual day. Bronfenbrenner himself argued against treating it as lab bad, field good. The point is to choose on purpose.\n\n## The evidence: lab and real life do not always agree\n\n**Research leans on the lab.** Bronfenbrenner's 1977 paper cites a survey by Larson of 902 child development studies published between 1972 and 1974. 76% used the experimental laboratory paradigm. Only 8% were observational studies of children in their everyday settings. Product research has its own version of this habit: the scheduled, moderated, screen-shared session.\n\n**What people say and what they do drift apart.** Paschal Sheeran's 2002 review pooled ten meta-analyses covering 422 studies. Across them, people's stated intentions accounted for an average of 28% of the variance in what they later did. That is a large effect by research standards, and it still leaves most of the behaviour unexplained. A study that only captures what participants say they would do, in a setting with no real stakes, is measuring the 28%.\n\n**Lab and field usability tests find different problems.** Henry Duh, Gerald Tan and Vivian Chen compared laboratory and real-world tests of the same mobile phone tasks in 2006. They reported significant differences in the frequency and severity of usability problems, in user behaviour and in how people felt about the device. Jesper Kjeldskov and Mikael Skov looked back on ten years of lab-versus-field studies in 2014 and concluded there was still no definite answer to which is better. The honest reading is that the setting changes what you find, and you can't assume the lab result transfers.\n\n**Remote does not have to mean less realistic.** In a 2007 CHI study, Morten Andreasen and colleagues found that remote synchronous testing, with the moderator in a different place, was virtually equivalent to a conventional lab test in the problems it found. The asynchronous methods they tested found fewer problems. Running research remotely, on the participant's own machine, can raise ecological validity without giving up much.\n\n**Most use happens on phones.** StatCounter data reported by Statista put mobile phones at about 62.5% of global web traffic in the second quarter of 2025, excluding tablets. If your sessions run on a desktop in an office, you are testing the minority experience for many products.\n\n## Where ecological validity breaks in user research\n\n**The setting is artificial.** Quiet rooms, fast Wi-Fi and full attention make tasks look easier than they are. Problems that only appear under interruption, poor signal or time pressure stay hidden.\n\n**The task is invented.** \"Imagine you want to book a trip to Lisbon\" asks people to role-play a goal they don't have. They make choices they wouldn't make with their own money and their own calendar. See [usability test tasks and scenarios](/docs/usability-test-tasks-scenarios) for ways to anchor tasks in real goals.\n\n**The data is fake.** A dashboard with tidy demo data hides the confusion of a real account with three years of half-finished projects.\n\n**The timing is wrong.** Asking about an experience weeks after it happened relies on memory, which smooths out the frustrating details. Asking at or close to the moment captures more of what actually happened. [Experience sampling](/docs/experience-sampling-method) and the [critical incident technique](/docs/critical-incident-technique) both exist to close this gap.\n\n**The participant knows they're being watched.** Observation changes behaviour, which is the [Hawthorne effect](/docs/hawthorne-effect-research). A visible moderator, a recording light and a room full of observers behind glass all add pressure that real use doesn't have.\n\n**The people are proxies.** Colleagues, friends of the team or panel members who match on demographics but have never done the job in question will give answers that sound right and aren't.\n\n## How to raise ecological validity\n\nYou don't have to move every study into the field. Use this checklist to make the conditions closer to real life wherever it matters most for the decision you need to make.\n\n![Checklist of eight ways to raise ecological validity in a user research study, from recruiting real users to noting artificial conditions in the report](/visuals/ecological-validity-research-checklist/ecological-validity-research-checklist.png \"Run it on your study plan before recruiting, and be honest about which boxes you left unticked\")\n\n1. **Recruit people who do the task for real.** Screen on behaviour (\"booked a work trip in the last month\"), not on demographics alone.\n2. **Use their device, account and data.** Remote sessions on the participant's own phone or laptop beat a lab machine for most product questions.\n3. **Ask about a specific recent time.** \"Tell me about the last time you...\" anchors answers in real events instead of a typical, idealised version.\n4. **Meet them where and when the task happens.** If people do it on the move or late at night, research it then.\n5. **Write tasks as goals from their week.** Let participants bring a real goal where you can, rather than handing them yours.\n6. **Reduce the feeling of being watched.** Fewer observers, shorter sessions and self-paced formats lower the pressure.\n7. **Compare what they said with what they did.** Check interview themes against product analytics or support tickets. See [stated vs revealed preferences](/docs/stated-vs-revealed-preferences).\n8. **Report the artificial parts.** Say plainly which conditions were not real, so readers know how far to trust each finding.\n\n## When lower ecological validity is fine\n\nNot every study needs maximum realism. Choose control when:\n\n- **You're isolating one variable.** Comparing two button labels is cleaner in a controlled setting where nothing else changes.\n- **You're early and exploring concepts.** A rough concept test can't be realistic because the thing doesn't exist yet. The aim is direction, not prediction.\n- **Safety or privacy rules it out.** Some settings, such as clinical or financial ones, can't be observed live.\n\nIn each case, the fix is to be explicit about it in the report and to plan a more realistic follow-up before anyone treats the result as a forecast.\n\n## Running research in the participant's own setting\n\nThe most reliable way to raise ecological validity is also the simplest: let people take part from wherever they normally are, on the device they normally use, at a time that suits their day.\n\n![A person at their kitchen table answering questions on their own phone, in the place they normally use the product](https://images.unsplash.com/photo-1758522484099-e0aa564f71ea?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwyfHxwZXJzb24lMjBwaG9uZSUyMGtpdGNoZW4lMjB0YWJsZSUyMGhvbWV8ZW58MXwwfHx8MTc5MjY0ODIxMHww&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Vitaly Gariev](https://unsplash.com/@silverkblack?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\nThat's what [contextual inquiry](/docs/contextual-inquiry) does in person, and what [remote usability testing](/docs/remote-usability-testing-guide) does on screen. Self-paced interviews go a step further by removing the scheduled call altogether. A participant can answer on their phone in the ten minutes after they've just used the product, while the experience is still fresh, instead of trying to recall it on a video call next Thursday.\n\nThe trade-off is less control over what happens in the session. You can't see their screen in a voice interview, and you can't step in when they misread a question. Good question design and follow-up probing make up much of that ground.\n\n## How Koji helps\n\nKoji runs AI-moderated interviews by voice or text. It can't make a lab session realistic, but it lets you take the interview to the participant instead of bringing the participant to you.\n\n- **Participants answer on their own device, in their own time.** The interview link opens on whatever phone or laptop they already use, with no call to schedule, so people can respond close to the moment that matters.\n- **Run the interview inside your product.** The Integrate screen gives you an iframe embed, a React component and a headless API, so you can ask about a task right where people do it instead of in an email days later.\n- **Voice, text or a phone call.** Participants choose voice or text on the welcome screen, and the call link lets someone type their number and get a phone call to run the interview by voice, for people who aren't at a desk.\n- **Ask about real recent events.** Open questions get AI follow-ups, from none up to three per question, so a vague \"it's usually fine\" gets a follow-up asking about the last time it wasn't.\n- **Screen on behaviour.** Screening questions run before every interview, whatever route a participant came in by, and anyone who doesn't qualify is turned away.\n- **See the evidence behind each finding.** Report findings show how many participants support them and link to the supporting responses, so you can check that a theme comes from real accounts, not hypotheticals.\n\n![Koji embed settings with an iframe code snippet, theme and height options and interview events](/email/shots/shot-embed.jpg \"Embedding the interview in your product lets people answer right where the experience happened\")\n\n## Related Resources\n\n- [Contextual inquiry](/docs/contextual-inquiry)\n- [The Hawthorne effect in research](/docs/hawthorne-effect-research)\n- [Remote usability testing guide](/docs/remote-usability-testing-guide)\n- [Stated vs revealed preferences](/docs/stated-vs-revealed-preferences)\n- [Qualitative research validity](/docs/qualitative-research-validity)\n- [Structured questions guide](/docs/structured-questions-guide)\n\n## Sources\n\n- Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. (2007). What happened to remote usability testing? An empirical study of three methods. _Proceedings of CHI 2007_, 1405–1414.\n- Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. _American Psychologist_, 32(7), 513–531.\n- Duh, H. B.-L., Tan, G. C. B., & Chen, V. H. (2006). Usability evaluation for mobile device: A comparison of laboratory and field tests. _Proceedings of MobileHCI 2006_, 181–186.\n- Kjeldskov, J., & Skov, M. B. (2014). Was it worth the hassle? Ten years of mobile HCI research discussions on lab and field evaluations. _Proceedings of MobileHCI 2014_, 43–52.\n- Sheeran, P. (2002). Intention–behavior relations: A conceptual and empirical review. _European Review of Social Psychology_, 12(1), 1–36.\n- StatCounter, via Statista (2025). Mobile share of global website traffic, Q2 2025.\n","category":"Research Methods","lastModified":"2026-10-22T05:50:10.698052+00:00","metaTitle":"Ecological Validity in User Research: Meaning and Examples (2026)","metaDescription":"What ecological validity means, how it differs from internal and external validity, what lab vs field studies show, and an 8-point checklist to raise it.","keywords":["ecological validity","ecological validity user research","ecological validity example","ecological vs external validity","lab vs field research","internal vs external validity","realistic usability testing"],"aiSummary":"Ecological validity is how closely a study's setting, tasks, materials and participants match real life. Bronfenbrenner (1977) criticised lab-bound research, citing a survey where 76% of 902 child development studies used the laboratory paradigm and 8% were observational. Sheeran (2002) found stated intentions explain about 28% of the variance in behaviour across 422 studies. Duh, Tan and Chen (2006) found lab and field mobile tests differed in the frequency and severity of usability problems, while Andreasen et al. (2007) found remote synchronous testing virtually equivalent to the lab. Mobile was about 62.5% of global web traffic in Q2 2025 (StatCounter via Statista). The guide compares internal, external and ecological validity, lists six ways studies lose realism, gives an eight-point checklist, explains when lower realism is acceptable and shows how Koji's self-paced voice and text interviews, in-product embed, call link, screening and evidence-linked findings help.","aiPrerequisites":["Basic familiarity with interviews or usability testing"],"aiLearningOutcomes":["Define ecological validity and where the term comes from","Tell ecological, internal and external validity apart","Summarise the evidence that lab and real-world results can differ","Spot the six ways user research studies lose realism","Use an eight-point checklist to raise ecological validity"],"aiDifficulty":"intermediate","aiEstimatedTime":"9 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}