All frameworks and cheat sheets

Comparison

Reliability vs validity in course evaluations

Comparison of reliability and validity in student evaluations of teaching: ratings are consistent, but barely track learning and respond to bias.
Consistent ratings are not the same as accurate ones, so read SET as a measure of student experience Source: Uttl, White and Gonzalez (2017), Boring, Ottoboni and Stark (2016)
Download PNG

A side-by-side comparison of reliability and validity as they apply to student evaluations of teaching (SET). Reliability asks whether ratings are consistent, and SET mostly are. Validity asks whether they measure teaching effectiveness, and the strongest meta-analytic evidence says they barely track learning while responding to instructor gender and grade expectations. Use it to explain to committees why a stable score can still be the wrong evidence. Koji for Education uses conversational interviews to capture the student experience that SET can validly report.

What's in this comparison

Reliability: is it consistent?

  • Same result on repeat measurement
  • Students in the same class tend to agree
  • Ratings stay similar year to year
  • A stable bias also passes this test

Validity: is it accurate?

  • Measures the thing it claims to measure
  • Ratings explain at most about 1% of learning variance
  • Ratings move more with gender and grade expectations
  • Valid for student experience, not teaching effectiveness

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/course-evaluation-reliability-vs-validity-comparison"><img src="https://www.koji.so/visuals/course-evaluation-reliability-vs-validity-comparison/course-evaluation-reliability-vs-validity-comparison.png" alt="Comparison of reliability and validity in student evaluations of teaching: ratings are consistent, but barely track learning and respond to bias." width="1200" height="642" /></a>
<p>Source: <a href="https://www.koji.so/visuals/course-evaluation-reliability-vs-validity-comparison">Reliability vs validity in course evaluations</a>, Koji</p>

Related

Comparison of a simulated population and real AI interviews on what each produces, who answers and where each is strongest

Simulated population vs real AI interviews

Comparison of a concierge MVP and a Wizard of Oz test: whether the customer knows the work is manual, how the product looks, and when each one fits.

Concierge MVP vs Wizard of Oz test

Comparison of the pause after asking a question with the pause after the participant answers, and what each one gets you in a research interview.

Two pauses in an interview: after the question and after the answer