Reference table
How often research measures lack validity evidence
Four findings on measurement practice in peer-reviewed psychology
A table of four findings collected by Jessica Flake and Eiko Fried on questionable measurement practices in psychology: measures without validity evidence, scales with no development history, scales reported without their source or item count, and ad hoc scales in the Many Labs 2 replication project. Researchers and product teams can use it to argue for checking construct validity before a metric goes on a dashboard.
Where this appears
What's in this reference table
| Finding | Source |
|---|---|
| 40 to 93% of measures in seven educational-behaviour journals lacked validity evidence | Barry et al. (2014) |
| 69% of 356 emotion measures cited no prior research or development process | Weidman, Steckler and Tracy (2017) |
| Of 433 JPSP scales (2014): 40% no source, 19% no item count, 9% no response scale | Flake, Pek and Hehman (2017) |
| 34 of 43 scales (79%) in Many Labs 2 looked ad hoc, with no validity information | Shaw et al. (2020) |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/measurement-validity-evidence-metascience-findings"><img src="https://www.koji.so/visuals/measurement-validity-evidence-metascience-findings/measurement-validity-evidence-metascience-findings.png" alt="Table of four metascience findings on measurement, including 40 to 93% of measures lacking validity evidence and 79% of Many Labs 2 scales looking ad hoc." width="1200" height="744" /></a>
<p>Source: <a href="https://www.koji.so/visuals/measurement-validity-evidence-metascience-findings">How often research measures lack validity evidence</a>, Koji</p>
