Research Automation: How to Build Real-Time Research Pipelines with Webhooks

The dream pipeline is simple: a user churns, an interview starts automatically, the AI digs into why, and the insight lands in Slack before your standup. You can build that on the Koji Headless API today. One honest caveat before the architecture: Koji does not have native outbound webhooks yet, so the completion signal comes from polling or the embed event rather than a push. Everything else in the pipeline works exactly as you would hope.

---

The pipeline at a glance

---

Stage 1: Trigger interviews from product events

Any event in your system can start an interview: a cancellation, a signup, a support ticket closing, an NPS detractor score. The start endpoint takes no project id (the API key determines the study), and the respondent object carries your CRM linkage:

// Example: trigger an exit interview when a user cancels
async function onSubscriptionCancelled(user) {
  const res = await fetch('https://www.koji.so/api/v1/interviews/start', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer ' + process.env.KOJI_API_KEY,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      respondent: {
        external_id: user.id,
        display_name: user.firstName,
        metadata: { plan: user.plan, cancelled_at: new Date().toISOString() },
      },
      mode: 'text',
    }),
  })

  const interview = await res.json()

  // Store the id so your completion worker can poll it later
  await db.pendingInterviews.insert({
    interview_id: interview.interview_id,
    external_id: user.id,
  })

  // Keep session_token too: your UI drives the conversation with it
  return interview
}

The two fields doing the heavy lifting:

• external_id ties the interview to your user record. It comes back in the read response, so the pipeline can join results to the right account.
• metadata is free-form JSON stored on the respondent. Stamp it with the trigger context (plan, cohort, event timestamp) so analysis lands with its context attached.

Full request reference in Starting Interviews via API.

---

Stage 2: Collect the conversation

Two ways to run the interview itself:

• Your own UI on the message API. Your client drives the conversation with the session_token, streaming AI replies over SSE. See Sending Messages via API and Completing Interviews via API.
• The embed widget. Drop the iframe into your app or a follow-up page and let it run the whole conversation. See Embed Widget Reference.

---

Stage 3: Detect completion and pull the analysis

Since there is no webhook push yet, completion detection is pull-based. The workhorse is a scheduled job over your pending interview ids:

// Runs on your server every minute (cron job, worker, or scheduled function).
// pendingIds holds interview ids you stored when you called the start endpoint.
const API_KEY = 'pk_live_wJalrXUtnFEMI4K7MDENGbPxRfiCYz2K'

async function checkPending(pendingIds) {
  for (const id of pendingIds) {
    const res = await fetch(`https://www.koji.so/api/v1/interviews/${id}`, {
      headers: { 'Authorization': `Bearer ${API_KEY}` },
    })
    if (!res.ok) continue

    const interview = await res.json()
    if (interview.status === 'completed' && interview.analysis) {
      await handleCompleted(interview) // your "webhook handler"
      // Remove id from your pending store so it is not processed twice
    }
  }
}

Key behaviors to build around:

• analysis is null until the interview is completed and the background analysis job has finished. Gating on status === 'completed' && analysis means one poll pass picks up fully processed interviews only.
• The read endpoint authenticates with the API key alone (interview:read permission, no session token), so the worker needs exactly one server-side key.
• Respect the per-key rate limit of 60 requests per minute when batching. See Rate Limits and CORS.

If your interviews run in the embed widget, the koji:interview_completed postMessage event gives you a faster in-browser signal you can forward to your backend. Use it to shave minutes off the loop, with polling as the safety net. Both patterns are compared in Webhook Setup.

---

Stage 4: Route the insights

Once the worker has a completed interview with analysis, routing is plain integration work. Slack is the classic first target:

// After your worker fetches a completed interview with analysis
const summary = interview.analysis?.summary ?? 'Analysis pending'

await fetch(process.env.SLACK_WEBHOOK_URL, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    text: `Interview ${interview.interview_id} completed ` +
      `(${interview.stats.message_count} messages)\n${summary}`,
  }),
})

Other destinations follow the same shape: upsert into your warehouse keyed on respondent.external_id, attach the summary to the CRM record that triggered the interview, or open a ticket when sentiment crosses a threshold. The read response gives you the full transcript, stats, and analysis in one payload, so a single fetch feeds every destination.

---

Putting it together

The whole lifecycle, runnable end to end in Node:

// Full interview lifecycle: start → message → complete → read results.
// Node 18+ (built-in fetch). Replace the key with your own.
const API_KEY = 'pk_live_wJalrXUtnFEMI4K7MDENGbPxRfiCYz2K'
const BASE = 'https://www.koji.so/api/v1'
const HEADERS = { 'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json' }

async function main() {
  // 1. Start
  const started = await fetch(`${BASE}/interviews/start`, {
    method: 'POST',
    headers: HEADERS,
    body: JSON.stringify({ respondent: { external_id: 'user_8271' }, mode: 'text' }),
  }).then(r => r.json())
  console.log('AI:', started.initial_message)

  const session = { 'X-Session-Token': started.session_token }

  // 2. One message exchange (SSE)
  const res = await fetch(`${BASE}/interviews/${started.interview_id}/message`, {
    method: 'POST',
    headers: { ...HEADERS, ...session },
    body: JSON.stringify({ content: 'I found onboarding confusing.' }),
  })
  let reply = ''
  const reader = res.body.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  while (true) {
    const { value, done } = await reader.read()
    if (done) break
    buffer += decoder.decode(value, { stream: true })
    const lines = buffer.split('\n')
    buffer = lines.pop()
    for (const line of lines) {
      if (!line.startsWith('data: ')) continue
      const frame = JSON.parse(line.slice(6))
      if (frame.type === 'chunk') reply += frame.content
    }
  }
  console.log('AI:', reply)

  // 3. Complete
  await fetch(`${BASE}/interviews/${started.interview_id}/complete`, {
    method: 'POST',
    headers: { ...HEADERS, ...session },
    body: JSON.stringify({ reason: 'natural' }),
  })

  // 4. Read results (analysis lands asynchronously)
  const result = await fetch(`${BASE}/interviews/${started.interview_id}`, {
    headers: HEADERS,
  }).then(r => r.json())
  console.log('Transcript messages:', result.stats.message_count)
}

main()

In production the stages split across your event handlers (stage 1), your UI or the embed (stage 2), and one scheduled worker (stages 3 and 4). The worker is the only new infrastructure the pipeline needs.

---

Native webhooks: coming soon

Outbound webhooks are on the roadmap. When they arrive, stage 3 becomes a push instead of a poll and end-to-end latency drops to near-real-time. Design for the swap now: keep completion handling in one function so the migration is a trigger change, not a rewrite.

---

FAQ

Is this pipeline real-time?

Near-real-time. With a one-minute poll cycle, insights land at most a couple of minutes after the respondent finishes. The embed completed event can shave that further for widget-based interviews.

Do I need one pipeline per study?

One worker can serve many studies, but API keys are study-scoped, so the worker holds one interview:read key per study and picks the right one per pending interview.

How do I connect an interview back to the user who triggered it?

Set respondent.external_id at start time. It comes back as respondent.external_id in the read response, ready to join against your user table.

What permissions should the pipeline's keys have?

The trigger path needs interview:start. The worker needs interview:read. If your own UI drives conversations, that client-side key needs interview:chat and interview:complete too. See API Key Permissions.