All frameworks and cheat sheets

Checklist

Audit a composite survey score in five minutes

Checklist for auditing a composite score: name one construct, list the items, check correlations, run a factor analysis, set the bar, look for ceilings.
Most composite scores fail the first check, before you compute anything.
Download PNGRead the full guide

A six-step checklist for any composite score you already report, such as a health score or trust index built from several survey questions. Name the single construct, list the items, check the average inter-item correlation, run a factor analysis, apply the reliability bar that fits the decision, and look for ceiling effects. Use it before you trust a Cronbach's alpha on a dashboard.

Where this appears

What's in this checklist

  • Name the one construct in a sentence. If you need the word and, you have two.
  • List the items word for word, and count them.
  • Compute the average inter-item correlation. Below about 0.3, alpha is mostly item count.
  • Run a factor analysis. More than one factor? Report them as separate scores.
  • Name the decision the score drives. For a specific account or user, use 0.90, not 0.70.
  • Check the distribution for ceilings. A score that can't go up can't show improvement.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/composite-score-reliability-audit-checklist"><img src="https://www.koji.so/visuals/composite-score-reliability-audit-checklist/composite-score-reliability-audit-checklist.png" alt="Checklist for auditing a composite score: name one construct, list the items, check correlations, run a factor analysis, set the bar, look for ceilings." width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/composite-score-reliability-audit-checklist">Audit a composite survey score in five minutes</a>, Koji</p>

Related

Table matching six questions about a survey scale to the right tool, such as omega, factor analysis or invariance testing, with alpha as the wrong tool.

What Cronbach's alpha can't tell you, and what can

Table of Nunnally's reliability thresholds: 0.70 for early-stage work, 0.80 for basic research, 0.90 minimum and 0.95 desirable when decisions rest on scores.

Nunnally's reliability thresholds, by how the score is used

Checklist for writing Likert scale statements: one specific claim, no double-barreled items, mixed wording, an odd number of points and every point labelled.

Likert scale statement checklist