Reference table
When to shuffle answer options in a course evaluation
A quick guide for evaluation designers on when to randomise the order of answer options. Primacy effects make options near the top of a list more likely to be chosen, mostly on long unordered lists. Shuffle those, never shuffle an ordered rating scale, and keep the same setting for any question you compare across cohorts. Koji can shuffle choice and ranking options for each student while keeping the last few options in place.
What's in this reference table
| Question type | Shuffle the options? | Why |
|---|---|---|
| Unordered single choice list | Yes | Spreads the pull of the top options evenly across the cohort |
| Unordered multi-select list | Yes, keep None of the above last | Options near the top get picked more when attention fades |
| Ordered rating scale | No | The order carries meaning, so shuffling only confuses students |
| Question you trend year on year | Use the same setting every cohort | Changing the format between cohorts creates change that is not real |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/when-to-shuffle-answer-options-course-evaluation"><img src="https://www.koji.so/visuals/when-to-shuffle-answer-options-course-evaluation/when-to-shuffle-answer-options-course-evaluation.png" alt="Table showing when to shuffle answer options in a course evaluation: yes for unordered lists, never for ordered rating scales, same setting for trended items" width="1200" height="744" /></a>
<p>Source: <a href="https://www.koji.so/visuals/when-to-shuffle-answer-options-course-evaluation">When to shuffle answer options in a course evaluation</a>, Koji</p>
