All frameworks and cheat sheets

Flowchart

Planning a quality assurance impact evaluation

Plan the evaluation with the QA procedure, not after it

Flowchart for planning a QA impact evaluation: write the outcome and mechanism, collect a baseline, find a comparison, then read answers for the mechanism.
Plan the evaluation with the procedure, not after it. Source: After Leiber, Stensaker and Harvey, Quality in Higher Education, 2015

A step-by-step decision path for QA offices introducing a new internal procedure, such as annual programme reviews, that they will later need to show made a difference. It follows the impact evaluation approach set out by Leiber, Stensaker and Harvey in 2015: a written outcome and mechanism, a baseline collected the same way as later waves, a comparison group from a staggered rollout, and open answers read for the mechanism. Koji can run the same interview-based course evaluation in each wave so the waves stay comparable.

What's in this flowchart

Start: A new QA procedure is planned

  1. Write the intended outcome. One sentence a student would recognise, such as clearer assessment criteria.
  2. Write the mechanism in three steps. What the procedure changes, how staff respond, what students experience.
  3. Is there a baseline collected the same way?
    • Yes: go to 4.
    • No: Collect one before launch. Same items, same mode, same timing as later waves. Then go to 4.
  4. Is there a comparison group?
    • Yes: go to 5.
    • No: Stagger the rollout. Programmes that join later act as the comparison. Then go to 5.
  5. Run each wave the same way. Report response rates and respondent mix with every result.
  6. Read open answers for the mechanism. Record what you find, even when it is not what you expected.
  7. Finish: Use the result to inform judgement. Evidence informs QA decisions alongside professional judgement.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/quality-assurance-impact-evaluation-planning-flowchart"><img src="https://www.koji.so/visuals/quality-assurance-impact-evaluation-planning-flowchart/quality-assurance-impact-evaluation-planning-flowchart.png" alt="Flowchart for planning a QA impact evaluation: write the outcome and mechanism, collect a baseline, find a comparison, then read answers for the mechanism." width="1200" height="1940" /></a>
<p>Source: <a href="https://www.koji.so/visuals/quality-assurance-impact-evaluation-planning-flowchart">Planning a quality assurance impact evaluation</a>, Koji</p>