All frameworks and cheat sheets

Flowchart

Did the teaching change work? A difference-in-differences check

Three questions to ask before crediting a redesign with a rise in scores

Flowchart for deciding whether a teaching change explains a rise in course evaluation scores: three questions, ending in crediting the change with its margin.
Run through it before reporting a rise to the programme board Source: after Campbell and Stanley (1963) and Card and Krueger (1994)

A decision flowchart for module leaders and quality teams who want to know whether a teaching change, not the year around it, raised a module's evaluation scores. It walks through the difference-in-differences logic: get a baseline, find a similar module that did not change, check that both moved together before, then subtract the comparison module's change. It follows Campbell and Stanley's warning about before-and-after designs and Card and Krueger's comparison-group method. Koji helps by asking the same questions in both modules each round.

What's in this flowchart

Start: Scores rose after a teaching change

  1. Do you have scores from before the change?
    • Yes: go to 2.
    • No: Collect a baseline first. Ask the same questions this term, before anything changes.
  2. Is there a similar module that did not change?
    • Yes: go to 3.
    • No: Find a comparison, or call it a hint. Before and after alone cannot separate the change from the year.
  3. Did both modules move in step in earlier years?
    • Yes: go to 4.
    • No: Report it as a hint, not a result. The comparison cannot stand in for your module.
  4. Subtract the comparison's change. Your module's rise minus the rise in modules that did not change.
  5. Check the gap against chance. A small gap on a small module is often noise.
  6. Finish: Credit the change, with its margin.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/credit-teaching-change-difference-in-differences-flowchart"><img src="https://www.koji.so/visuals/credit-teaching-change-difference-in-differences-flowchart/credit-teaching-change-difference-in-differences-flowchart.png" alt="Flowchart for deciding whether a teaching change explains a rise in course evaluation scores: three questions, ending in crediting the change with its margin." width="1200" height="1618" /></a>
<p>Source: <a href="https://www.koji.so/visuals/credit-teaching-change-difference-in-differences-flowchart">Did the teaching change work? A difference-in-differences check</a>, Koji</p>