{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-07-31T02:08:35.497Z"},"content":[{"type":"documentation","id":"7ad22fee-9a2d-4ef7-86cd-24327cf9598e","slug":"developer-experience-research","title":"Developer Experience Research: How to Measure DevEx With Interviews, Not Just Surveys","url":"https://www.koji.so/docs/developer-experience-research","summary":"Developer experience research is the qualitative layer beneath DevEx metrics. DORA measures delivery throughput and stability, SPACE provides a dimensional model, and the DX Core 4 (Laura Tacho and Abi Noda, with the DORA, SPACE and DevEx authors, applied across 300+ organisations with reported 3-12% efficiency gains and ~14% more R&D time on features) unifies them into speed, effectiveness, quality and business impact. None explains mechanism. The DevEx dimensions — feedback loops, cognitive load, flow state — are the interview-friendly layer. Quarterly surveys underperform because response rates decay, scores lack mechanisms, free-text collapses, and fear of attribution suppresses candour. The central 2026 question is whether AI coding tools are net positive: the 2025 Stack Overflow Developer Survey found 84% adoption, only 3% high trust, 46% distrusting accuracy versus 33% trusting, trust down 11 points to 29%, 66% frustrated by nearly-right output and 45% reporting AI code takes longer to debug. Koji's six structured question types map to DevEx measurement: scale for trackable scores, yes_no for blocking gates, single_choice for attribution, multiple_choice for friction inventory, ranking for forced platform prioritisation, and open_ended with AI follow-up probing for mechanism. Text mode suits engineers better than voice. Cadence should be quarterly full instrument plus continuous triggered short rounds. Anonymity, separation from performance management, non-leading questions about mandated tooling, and publicly closing the loop are prerequisites.","content":"**Answer first:** Developer experience research is the qualitative layer underneath your DevEx metrics. Frameworks like DORA, SPACE and the DX Core 4 tell you *what* your engineering organisation's numbers are; they cannot tell you *why* deploy frequency dropped, *which* part of the local build makes people give up, or *whether* the AI coding tools you paid for actually removed work or just moved it. That requires talking to engineers — and engineers are the audience least likely to complete a survey and least available for a scheduled call. Async AI-moderated interviews solve both problems: a developer answers at 11pm between builds, and the AI asks the follow-up question a Likert scale never could.\n\n> **Researching developers as customers of your product?** That is a different job — see [User Research for Developer Tools](/blog/user-research-for-developer-tools-2026). This guide is about measuring the experience of engineers *inside your own organisation*.\n\n## What the metrics frameworks do and do not cover\n\nThree frameworks dominate the space, and they stack rather than compete:\n\n| Framework | What it measures | What it structurally cannot tell you |\n|---|---|---|\n| **DORA** | Delivery throughput and stability — deploy frequency, lead time, change failure rate, failed-deployment recovery | Why a number moved. It observes the system, not the people in it |\n| **SPACE** | A dimensional model: satisfaction, performance, activity, communication, efficiency | What to actually put in each dimension. It is a framework for defining metrics, not a metric set |\n| **DX Core 4** | Four dimensions — speed, effectiveness, quality, business impact — unifying DORA, SPACE and DevEx into one set of primary and secondary metrics | The mechanism behind any single score |\n\nThe DX Core 4, developed by Laura Tacho and Abi Noda with the authors of DORA, SPACE and DevEx, has been applied across more than 300 organisations, with reported outcomes in the range of 3–12% improvement in engineering efficiency and around 14% more R&D time spent on feature development. Its explicit design goal is to combine system telemetry with self-reported experience, because neither alone predicts outcomes.\n\nAnd that self-reported half is where most organisations quietly fail. They run a quarterly survey, get a 3.2 out of 5 on \"local development environment,\" and have no idea what to fix.\n\n## The DevEx dimensions worth interviewing about\n\nThe DevEx framework — feedback loops, cognitive load, and flow state — is the most interview-friendly of the three, because each dimension is a lived experience rather than a system property.\n\n| Dimension | The metric tells you | The interview tells you |\n|---|---|---|\n| **Feedback loops** | Median CI time is 14 minutes | That people context-switch at minute 3, lose the thread, and the real cost is the re-entry, not the 14 minutes |\n| **Cognitive load** | Onboarding to first PR takes 9 days | That day 4 is spent guessing which of three deployment paths is current, because two runbooks contradict each other |\n| **Flow state** | 41% report frequent uninterrupted time | That the interruptions are on-call pages for alerts nobody has tuned in a year |\n\nEvery one of those right-hand answers is a fixable work item. None of them is derivable from the left-hand number.\n\n## Why the quarterly DevEx survey underperforms\n\nFour failure modes, all common:\n\n1. **Response rates decay.** Engineers are the most survey-fatigued population in the company, and the survey that took twelve minutes last quarter gets abandoned this quarter. Declining response rates are an industry-wide pattern, not a local one — see [why survey response rates are falling](/docs/survey-response-rates-declining).\n2. **Scores without mechanisms.** A Likert score is a symptom. Acting on it requires a diagnosis you did not collect.\n3. **Free-text fields collapse.** The one open box gets \"build is slow\" from forty people and nothing you can scope.\n4. **Fear of attribution.** If a developer suspects their manager can identify the response, the honest answer never appears. Anonymity is not a nice-to-have in internal research — it is the precondition for validity.\n\nThe other option — scheduling 1:1 interviews — produces excellent data and does not scale. A staff engineer's calendar is the scarcest resource in the building, and a research programme that depends on twelve of them agreeing to a 45-minute slot runs twice a year at best.\n\n## The AI-tooling question nobody has a clean answer to\n\nIf you measure one thing qualitatively this year, measure this. The 2025 Stack Overflow Developer Survey found AI tool adoption at a record 84% while trust in those tools hit an all-time low: only 3% of developers said they highly trust AI output, 46% actively distrust its accuracy against 33% who trust it, and overall trust fell 11 percentage points to 29%. The top frustration, cited by 66%, was \"AI solutions that are almost right, but not quite\" — followed by 45% reporting that debugging AI-generated code takes longer than writing it.\n\nRead those numbers together and you get the central DevEx question of 2026: your adoption dashboard is green, your satisfaction score is flat, and nobody can tell you whether the tools are net positive. Adoption is not value. The gap between them is only visible in conversation — and the specific probe that surfaces it is \"walk me through the last time an AI suggestion cost you more time than it saved.\"\n\nThis is a research problem, not a telemetry problem. For the wider version of it across all functions, see [employee AI adoption research](/docs/employee-ai-adoption-research); for judging AI output quality rigorously, see [human evaluation of AI outputs](/docs/human-evaluation-ai-outputs).\n\n## Designing the study: map dimensions to question types\n\nKoji's six [structured question types](/docs/structured-questions-guide) map onto DevEx measurement unusually cleanly, which lets a single instrument carry both the trackable number and the reason behind it:\n\n| Question type | DevEx use |\n|---|---|\n| `scale` | The trackable score — satisfaction with local dev, CI, deploys, documentation. Anchor it so \"You said 4 — what would make it a 7?\" follows automatically |\n| `yes_no` | Binary gates: \"Did you have to ask someone to unblock a deploy this week?\" |\n| `single_choice` | Attribution: which stage of the loop costs you most time |\n| `multiple_choice` | Friction inventory: every place work stalled this sprint |\n| `ranking` | Forced prioritisation across candidate platform investments — far more decision-useful than five separate importance scales |\n| `open_ended` | The mechanism, with AI follow-up probing to depth 2 or 3 |\n\nThat last row is the one a survey tool cannot replicate. When a developer says \"the test suite is flaky,\" Koji's AI interviewer asks which suite, what they do when it fails, and how long the workaround takes — the three facts a platform team needs to actually schedule work. Follow-up depth is configurable per question, so your scale items stay fast and your diagnostic items go deep.\n\n### A ten-question DevEx instrument\n\n1. `scale` — Overall, how would you rate your development experience this quarter? (1–10, anchored)\n2. `open_ended` — What is the single biggest thing slowing you down right now? (probe: frequency, workaround, time cost)\n3. `single_choice` — Where does most of your lost time go? (local setup / CI / code review / deploys / debugging / meetings / unclear requirements)\n4. `scale` — How long does it typically take to get a change reviewed? (bucketed)\n5. `yes_no` — Were you blocked for more than half a day this sprint waiting on someone else?\n6. `open_ended` — Describe the last time you lost flow. What pulled you out? (probe: recovery cost)\n7. `multiple_choice` — Which of these did you have to ask a human for because documentation did not answer it?\n8. `ranking` — Rank these platform investments by impact on your week.\n9. `scale` — How much do you trust the output of the AI tools you use daily? (1–10)\n10. `open_ended` — Tell me about a time an AI suggestion cost you more time than it saved. (probe: detection, fix, whether you still use it)\n\nTen questions, roughly nine minutes conversational, and it produces both a quarter-over-quarter trend line and a ranked list of fixable causes.\n\n## Anonymity, ethics, and not Goodharting your engineers\n\nInternal research on productivity is politically loaded, and mishandling it poisons the well for years.\n\n- **Never report individual-level data.** Aggregate to team level at minimum, and only where teams are large enough that a response cannot be inferred. Koji's async link-based interviews mean no moderator sees who said what in real time, and you can run studies without collecting identifying fields at all — see [anonymous employee research](/docs/anonymous-employee-research-ai-interviews).\n- **Separate DevEx measurement from performance management, explicitly and in writing.** The moment developers suspect DevEx data feeds a review cycle, you are measuring impression management.\n- **Do not ask leading questions about mandated tooling.** \"How much has our new AI assistant improved your productivity?\" produces the answer leadership wants. \"Walk me through your last session with it\" produces the truth.\n- **Close the loop publicly.** Publish what you heard and what you changed. Engineers are a high-scepticism audience: one round of research with no visible consequence guarantees a lower response rate next time.\n\n## Cadence: quantitative quarterly, qualitative continuously\n\nThe standard mistake is running everything at once, twice a year. A better shape:\n\n| Cadence | Instrument | Purpose |\n|---|---|---|\n| **Quarterly** | Full DevEx instrument, whole engineering org | Trend lines, DX Core 4 self-reported inputs, board-level reporting |\n| **Continuous** | Two to four questions, triggered | Diagnose specific friction while it is fresh — after onboarding, after an incident, after a platform migration |\n| **Ad hoc** | Targeted round, one team | Investigate a metric that moved and nobody can explain |\n\nBecause Koji interviews are async and link-based, the triggered rounds are the cheap ones: a new hire gets an interview link at day 30, an incident participant gets one the day after the postmortem, and the responses land in the same structured schema as the quarterly instrument so they aggregate together. Automate the triggers with [webhooks](/docs/research-automation-webhooks) or drive the whole thing from your editor via [Koji's MCP server](/docs/mcp-setup-cursor) — fifteen tools covering study creation, publishing, data retrieval and reporting, which means a platform team can run DevEx research from the same terminal they run everything else from.\n\nCosts stay predictable: text interviews consume 1 credit each, voice interviews 3, so a 40-person quarterly text round is 40 credits — inside the Interviews plan's monthly allowance of 79. Koji's quality gate only charges for conversations scoring 3 or above out of 5, so abandoned or non-substantive sessions do not burn budget.\n\n## Voice or text?\n\nFor engineers, default to text. It fits the way they already work, can be completed in fragments between builds, and produces clean transcripts of technical detail — tool names, error strings, branch names — that voice transcription mangles. Reserve [voice interviews](/docs/ai-voice-interviews) for narrative sessions where you want the emotional register: incident retrospectives, onboarding stories, migration frustration. The trade-offs are laid out in [voice vs text interviews](/docs/voice-vs-text-interviews).\n\n## Turning findings into platform work\n\nDevEx research that ends in a slide deck changes nothing. Each finding should become a scoped work item with frequency and severity attached, filed where the platform team already works — see [research handoff to design and engineering](/docs/research-handoff-design-engineering) for the packet format, and the [Linear integration](/docs/linear-research-integration) for filing evidence-linked tickets straight from interview insights.\n\n## Frequently asked questions\n\n**What is developer experience research?**\nIt is the qualitative study of how engineers actually work — the feedback loops, cognitive load, and interruptions that determine how much of their day becomes shipped software. It complements DevEx metrics frameworks like DORA, SPACE and the DX Core 4 by explaining the mechanisms behind the numbers those frameworks report.\n\n**How is DevEx research different from a developer productivity survey?**\nA survey collects scores; DevEx research collects scores plus causes. In practice the difference is follow-up: when a developer rates code review 4 out of 10, a survey records a 4 while an interview establishes that reviews stall for two days waiting on one overloaded reviewer — which is an actionable staffing fix rather than a number on a dashboard.\n\n**How often should we run DevEx research?**\nRun the full instrument quarterly for trend lines, and short triggered rounds continuously — after onboarding, after incidents, after platform migrations. Quarterly-only programmes miss the causes because they ask months after the friction happened.\n\n**How do we get engineers to actually respond?**\nKeep it under ten minutes, make it async so it fits between builds rather than on a calendar, guarantee anonymity credibly, and publish what changed as a result of the last round. Async AI-moderated interviews outperform both static surveys and scheduled 1:1s here, because they remove the two things engineers object to most: pointless Likert grids and meetings.\n\n**Can DevEx research measure whether AI coding tools are helping?**\nYes, and it is currently the best available method. Adoption telemetry shows usage, not value — the 2025 Stack Overflow survey found 84% adoption alongside only 3% high trust and 66% citing \"almost right, but not quite\" as their top frustration. Asking developers to walk through specific sessions where AI cost them time surfaces the net effect that dashboards hide.\n\n**Is it safe to run internal research on productivity?**\nOnly if you separate it from performance management explicitly, aggregate results to team level, and never report individual responses. Anonymous async interviews with no moderator present and no identifying fields collected are the safest configuration, and they materially improve candour.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide) — the six question types and how each aggregates\n- [Employee AI Adoption Research](/docs/employee-ai-adoption-research) — the same question across every function, not just engineering\n- [Anonymous Employee Research with AI Interviews](/docs/anonymous-employee-research-ai-interviews) — how to protect candour on internal studies\n- [Human Evaluation of AI Outputs](/docs/human-evaluation-ai-outputs) — judging whether AI output is actually good\n- [Voice vs Text Interviews](/docs/voice-vs-text-interviews) — choosing the mode for a technical audience\n- [Research Handoff to Design and Engineering](/docs/research-handoff-design-engineering) — turning findings into platform tickets\n- [Connect Koji to Cursor (MCP)](/docs/mcp-setup-cursor) — running research from the editor","category":"Use Cases","lastModified":"2026-07-30T03:21:11.273045+00:00","metaTitle":"Developer Experience Research: Measuring DevEx With Interviews (2026)","metaDescription":"DORA and the DX Core 4 give you the score, not the reason. How to run the qualitative DevEx layer — instrument design, anonymity, cadence — with async AI interviews engineers finish.","keywords":["developer experience research","developer experience survey","how to measure developer experience","devex research","DX Core 4","DORA metrics qualitative","engineering productivity survey","developer productivity research","internal developer platform research"],"aiSummary":"Developer experience research is the qualitative layer beneath DevEx metrics. DORA measures delivery throughput and stability, SPACE provides a dimensional model, and the DX Core 4 (Laura Tacho and Abi Noda, with the DORA, SPACE and DevEx authors, applied across 300+ organisations with reported 3-12% efficiency gains and ~14% more R&D time on features) unifies them into speed, effectiveness, quality and business impact. None explains mechanism. The DevEx dimensions — feedback loops, cognitive load, flow state — are the interview-friendly layer. Quarterly surveys underperform because response rates decay, scores lack mechanisms, free-text collapses, and fear of attribution suppresses candour. The central 2026 question is whether AI coding tools are net positive: the 2025 Stack Overflow Developer Survey found 84% adoption, only 3% high trust, 46% distrusting accuracy versus 33% trusting, trust down 11 points to 29%, 66% frustrated by nearly-right output and 45% reporting AI code takes longer to debug. Koji's six structured question types map to DevEx measurement: scale for trackable scores, yes_no for blocking gates, single_choice for attribution, multiple_choice for friction inventory, ranking for forced platform prioritisation, and open_ended with AI follow-up probing for mechanism. Text mode suits engineers better than voice. Cadence should be quarterly full instrument plus continuous triggered short rounds. Anonymity, separation from performance management, non-leading questions about mandated tooling, and publicly closing the loop are prerequisites.","aiPrerequisites":["An engineering organisation with at least one existing DevEx or delivery metric","Agreement from engineering leadership that results will not feed performance reviews"],"aiLearningOutcomes":["Explain what DORA, SPACE and the DX Core 4 can and cannot tell you","Interview against the three DevEx dimensions: feedback loops, cognitive load, flow state","Build a ten-question DevEx instrument using six structured question types","Measure whether AI coding tools are net positive rather than merely adopted","Protect candour with anonymity and separation from performance management","Set a cadence of quarterly full instrument plus continuous triggered rounds"],"aiDifficulty":"intermediate","aiEstimatedTime":"12 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}