All frameworks and cheat sheets

Flowchart

Does your course evaluation need an attention check?

Flowchart for deciding whether a course evaluation needs an attention check: three questions, two side steps, ending in published counts of flagged answers.
Most course evaluations are better served by prevention and transparent flags than by a trap item. Source: Koji synthesis of Oppenheimer et al. 2009, Berinsky et al. 2014, Hauser and Schwarz 2015, Anduiza and Galais 2017
Download PNG

A decision flowchart for quality officers and survey leads deciding whether to add an attention check, such as an instructed-response or trap item, to a course evaluation. It follows the research on instructional manipulation checks: one screener is a weak measure, checks change how people answer, and dropping everyone who fails biases descriptive results. Use it before each survey cycle. Koji runs evaluations as short interviews that flag low-effort answers instead of hiding them.

What's in this flowchart

Start: Planning a course evaluation

  1. Is the survey describing what students experienced?
    • Yes: go to 2.
    • No: Use several pre-treatment checks. Experiments gain power from screening. Set the rules before data comes in.
  2. Can students finish the form in a few minutes?
    • Yes: go to 3.
    • No: Cut and clarify items first. Long forms invite clicking through. Shorten before adding screeners. Then go to 3.
  3. Skip the trap item. One screener is a weak measure and changes how students read later items.
  4. Flag low-effort answers. Use response time, thin follow-up answers or ratings that clash with comments. Keep every response.
  5. Do the flagged answers change the result?
    • No: go to 6.
    • Yes: Report both versions. Show results with and without flagged answers, and say why. Then go to 6.
  6. Finish: Publish counts of flagged answers. Put the number next to every course mean.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/attention-check-course-evaluation-decision-flowchart"><img src="https://www.koji.so/visuals/attention-check-course-evaluation-decision-flowchart/attention-check-course-evaluation-decision-flowchart.png" alt="Flowchart for deciding whether a course evaluation needs an attention check: three questions, two side steps, ending in published counts of flagged answers." width="1200" height="1686" /></a>
<p>Source: <a href="https://www.koji.so/visuals/attention-check-course-evaluation-decision-flowchart">Does your course evaluation need an attention check?</a>, Koji</p>

Related

Flowchart for low activation: three questions that lead to fixing who you attract, rewriting the first screen or removing a blocking step.

Wrong audience or wrong first session?

Flowchart for triaging accounts after a price increase: three questions, one side step to brief the signer, ending in a weekly account review.

After a price increase: which accounts get a call?

Flowchart for deciding after customer discovery: three questions on the problem, workaround spend and commitments, ending in building the smallest version.

Build, pivot or stop after customer discovery