Reference table
Six kinds of validity evidence, and a cheap test for each
How to check a metric measures what its name says
A practical table of six kinds of evidence for construct validity: face, content, convergent, discriminant, criterion and response process. For each it gives the question it answers and a test a product team can run without a statistics team, such as correlating headline metrics with each other or asking respondents what they thought a question meant. Koji follows a scale question with an open probe in the same interview, which gives response-process evidence as you collect the number.
Where this appears
What's in this reference table
| Evidence type | The question it answers | A cheap version to run |
|---|---|---|
| Face validity | Does it look like it measures the construct? | Show the items to five colleagues and ask what they think is being measured |
| Content validity | Does it cover the whole construct? | List the facets of your definition and check each has at least one item |
| Convergent | Does it relate to what it should? | Correlate with a behavioural outcome you already log |
| Discriminant | Is it distinct from adjacent constructs? | Correlate your headline metrics with each other. This is the test that matters |
| Criterion or predictive | Does it predict what it should? | Check whether last quarter scores predicted this quarter behaviour |
| Response process | Do respondents read items as intended? | Cognitive interviews: ask people what they thought the question meant |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/construct-validity-evidence-types-cheap-tests"><img src="https://www.koji.so/visuals/construct-validity-evidence-types-cheap-tests/construct-validity-evidence-types-cheap-tests.png" alt="Table of six kinds of construct validity evidence, from face validity to response process, with the question each answers and a cheap way to test it." width="1200" height="952" /></a>
<p>Source: <a href="https://www.koji.so/visuals/construct-validity-evidence-types-cheap-tests">Six kinds of validity evidence, and a cheap test for each</a>, Koji</p>
