Developer Experience Research: How to Measure DevEx With Interviews, Not Just Surveys

Answer first: Developer experience research is the qualitative layer underneath your DevEx metrics. Frameworks like DORA, SPACE and the DX Core 4 tell you what your engineering organisation's numbers are; they cannot tell you why deploy frequency dropped, which part of the local build makes people give up, or whether the AI coding tools you paid for actually removed work or just moved it. That requires talking to engineers — and engineers are the audience least likely to complete a survey and least available for a scheduled call. Async AI-moderated interviews solve both problems: a developer answers at 11pm between builds, and the AI asks the follow-up question a Likert scale never could.

Researching developers as customers of your product? That is a different job — see User Research for Developer Tools. This guide is about measuring the experience of engineers inside your own organisation.

What the metrics frameworks do and do not cover

Three frameworks dominate the space, and they stack rather than compete:

The DX Core 4, developed by Laura Tacho and Abi Noda with the authors of DORA, SPACE and DevEx, has been applied across more than 300 organisations, with reported outcomes in the range of 3–12% improvement in engineering efficiency and around 14% more R&D time spent on feature development. Its explicit design goal is to combine system telemetry with self-reported experience, because neither alone predicts outcomes.

And that self-reported half is where most organisations quietly fail. They run a quarterly survey, get a 3.2 out of 5 on "local development environment," and have no idea what to fix.

The DevEx dimensions worth interviewing about

The DevEx framework — feedback loops, cognitive load, and flow state — is the most interview-friendly of the three, because each dimension is a lived experience rather than a system property.

Every one of those right-hand answers is a fixable work item. None of them is derivable from the left-hand number.

Why the quarterly DevEx survey underperforms

Four failure modes, all common:

1. Response rates decay. Engineers are the most survey-fatigued population in the company, and the survey that took twelve minutes last quarter gets abandoned this quarter. Declining response rates are an industry-wide pattern, not a local one — see why survey response rates are falling.
2. Scores without mechanisms. A Likert score is a symptom. Acting on it requires a diagnosis you did not collect.
3. Free-text fields collapse. The one open box gets "build is slow" from forty people and nothing you can scope.
4. Fear of attribution. If a developer suspects their manager can identify the response, the honest answer never appears. Anonymity is not a nice-to-have in internal research — it is the precondition for validity.

The other option — scheduling 1:1 interviews — produces excellent data and does not scale. A staff engineer's calendar is the scarcest resource in the building, and a research programme that depends on twelve of them agreeing to a 45-minute slot runs twice a year at best.

The AI-tooling question nobody has a clean answer to

If you measure one thing qualitatively this year, measure this. The 2025 Stack Overflow Developer Survey found AI tool adoption at a record 84% while trust in those tools hit an all-time low: only 3% of developers said they highly trust AI output, 46% actively distrust its accuracy against 33% who trust it, and overall trust fell 11 percentage points to 29%. The top frustration, cited by 66%, was "AI solutions that are almost right, but not quite" — followed by 45% reporting that debugging AI-generated code takes longer than writing it.

Read those numbers together and you get the central DevEx question of 2026: your adoption dashboard is green, your satisfaction score is flat, and nobody can tell you whether the tools are net positive. Adoption is not value. The gap between them is only visible in conversation — and the specific probe that surfaces it is "walk me through the last time an AI suggestion cost you more time than it saved."

This is a research problem, not a telemetry problem. For the wider version of it across all functions, see employee AI adoption research; for judging AI output quality rigorously, see human evaluation of AI outputs.

Designing the study: map dimensions to question types

Koji's six structured question types map onto DevEx measurement unusually cleanly, which lets a single instrument carry both the trackable number and the reason behind it:

That last row is the one a survey tool cannot replicate. When a developer says "the test suite is flaky," Koji's AI interviewer asks which suite, what they do when it fails, and how long the workaround takes — the three facts a platform team needs to actually schedule work. Follow-up depth is configurable per question, so your scale items stay fast and your diagnostic items go deep.

A ten-question DevEx instrument

1. scale — Overall, how would you rate your development experience this quarter? (1–10, anchored)
2. open_ended — What is the single biggest thing slowing you down right now? (probe: frequency, workaround, time cost)
3. single_choice — Where does most of your lost time go? (local setup / CI / code review / deploys / debugging / meetings / unclear requirements)
4. scale — How long does it typically take to get a change reviewed? (bucketed)
5. yes_no — Were you blocked for more than half a day this sprint waiting on someone else?
6. open_ended — Describe the last time you lost flow. What pulled you out? (probe: recovery cost)
7. multiple_choice — Which of these did you have to ask a human for because documentation did not answer it?
8. ranking — Rank these platform investments by impact on your week.
9. scale — How much do you trust the output of the AI tools you use daily? (1–10)
10. open_ended — Tell me about a time an AI suggestion cost you more time than it saved. (probe: detection, fix, whether you still use it)

Ten questions, roughly nine minutes conversational, and it produces both a quarter-over-quarter trend line and a ranked list of fixable causes.

Anonymity, ethics, and not Goodharting your engineers

Internal research on productivity is politically loaded, and mishandling it poisons the well for years.

• Never report individual-level data. Aggregate to team level at minimum, and only where teams are large enough that a response cannot be inferred. Koji's async link-based interviews mean no moderator sees who said what in real time, and you can run studies without collecting identifying fields at all — see anonymous employee research.
• Separate DevEx measurement from performance management, explicitly and in writing. The moment developers suspect DevEx data feeds a review cycle, you are measuring impression management.
• Do not ask leading questions about mandated tooling. "How much has our new AI assistant improved your productivity?" produces the answer leadership wants. "Walk me through your last session with it" produces the truth.
• Close the loop publicly. Publish what you heard and what you changed. Engineers are a high-scepticism audience: one round of research with no visible consequence guarantees a lower response rate next time.

Cadence: quantitative quarterly, qualitative continuously

The standard mistake is running everything at once, twice a year. A better shape:

Because Koji interviews are async and link-based, the triggered rounds are the cheap ones: a new hire gets an interview link at day 30, an incident participant gets one the day after the postmortem, and the responses land in the same structured schema as the quarterly instrument so they aggregate together. Automate the triggers with webhooks or drive the whole thing from your editor via Koji's MCP server — fifteen tools covering study creation, publishing, data retrieval and reporting, which means a platform team can run DevEx research from the same terminal they run everything else from.

Costs stay predictable: text interviews consume 1 credit each, voice interviews 3, so a 40-person quarterly text round is 40 credits — inside the Interviews plan's monthly allowance of 79. Koji's quality gate only charges for conversations scoring 3 or above out of 5, so abandoned or non-substantive sessions do not burn budget.

Voice or text?

For engineers, default to text. It fits the way they already work, can be completed in fragments between builds, and produces clean transcripts of technical detail — tool names, error strings, branch names — that voice transcription mangles. Reserve voice interviews for narrative sessions where you want the emotional register: incident retrospectives, onboarding stories, migration frustration. The trade-offs are laid out in voice vs text interviews.

Turning findings into platform work

DevEx research that ends in a slide deck changes nothing. Each finding should become a scoped work item with frequency and severity attached, filed where the platform team already works — see research handoff to design and engineering for the packet format, and the Linear integration for filing evidence-linked tickets straight from interview insights.

Frequently asked questions

What is developer experience research? It is the qualitative study of how engineers actually work — the feedback loops, cognitive load, and interruptions that determine how much of their day becomes shipped software. It complements DevEx metrics frameworks like DORA, SPACE and the DX Core 4 by explaining the mechanisms behind the numbers those frameworks report.

How is DevEx research different from a developer productivity survey? A survey collects scores; DevEx research collects scores plus causes. In practice the difference is follow-up: when a developer rates code review 4 out of 10, a survey records a 4 while an interview establishes that reviews stall for two days waiting on one overloaded reviewer — which is an actionable staffing fix rather than a number on a dashboard.

How often should we run DevEx research? Run the full instrument quarterly for trend lines, and short triggered rounds continuously — after onboarding, after incidents, after platform migrations. Quarterly-only programmes miss the causes because they ask months after the friction happened.

How do we get engineers to actually respond? Keep it under ten minutes, make it async so it fits between builds rather than on a calendar, guarantee anonymity credibly, and publish what changed as a result of the last round. Async AI-moderated interviews outperform both static surveys and scheduled 1:1s here, because they remove the two things engineers object to most: pointless Likert grids and meetings.

Can DevEx research measure whether AI coding tools are helping? Yes, and it is currently the best available method. Adoption telemetry shows usage, not value — the 2025 Stack Overflow survey found 84% adoption alongside only 3% high trust and 66% citing "almost right, but not quite" as their top frustration. Asking developers to walk through specific sessions where AI cost them time surfaces the net effect that dashboards hide.

Is it safe to run internal research on productivity? Only if you separate it from performance management explicitly, aggregate results to team level, and never report individual responses. Anonymous async interviews with no moderator present and no identifying fields collected are the safest configuration, and they materially improve candour.

Related Resources

• Structured Questions Guide — the six question types and how each aggregates
• Employee AI Adoption Research — the same question across every function, not just engineering
• Anonymous Employee Research with AI Interviews — how to protect candour on internal studies
• Human Evaluation of AI Outputs — judging whether AI output is actually good
• Voice vs Text Interviews — choosing the mode for a technical audience
• Research Handoff to Design and Engineering — turning findings into platform tickets
• Connect Koji to Cursor (MCP) — running research from the editor