Flowchart
Open or closed course evaluation question?
A decision path based on split-ballot evidence that question format changes the answers you get
A decision flowchart for quality officers and programme directors choosing whether a course evaluation question should be open or closed. It follows the split-ballot finding by Reja and colleagues (2003) that closed lists pull answers toward the options named, while open questions catch more variety but more missing data. Use it when reviewing a questionnaire. Koji can pair a closed answer with an open follow-up in one interview.
What's in this flowchart
Start: You are adding an evaluation question
- Can you list the likely answers in advance?
- Yes: go to 2.
- No: Run it open in a pilot first. Code what students write and build the list from it. Then go to 2.
- Do you need counts to compare across courses?
- Yes: go to 3.
- No: Keep it open, with a specific prompt. Ask about one concrete part of the course.
- Write the closed list from real answers. Add an Other option so new problems can still surface.
- Will someone act on the reasons for a choice?
- No: go to 5.
- Yes: Add one open follow-up. Ask why, straight after the closed answer. Then go to 5.
- Finish: Report counts and list the Other answers.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/open-or-closed-course-evaluation-question-flowchart"><img src="https://www.koji.so/visuals/open-or-closed-course-evaluation-question-flowchart/open-or-closed-course-evaluation-question-flowchart.png" alt="Flowchart for choosing between an open and a closed course evaluation question: three questions, two side steps and one exit, ending in reported counts." width="1200" height="1293" /></a>
<p>Source: <a href="https://www.koji.so/visuals/open-or-closed-course-evaluation-question-flowchart">Open or closed course evaluation question?</a>, Koji</p>
