The lead article in the June 2010 edition of the Journal of Political Economy is
Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors | ||
Scott E. Carrell and James E West |
Student evaluations may not be a good signal of teaching quality because
“Professors can inflate grades or reduce academic content to elevate student evaluations.”
The authors argue that if a student takes Calculus I, say, their performance in Calculus II is a good signal of how well they learned the material in Calculus I. So their study:
“uses a unique panel data set from the United States Air Force Academy (USAFA) in which students
are randomly assigned to professors over a wide variety of standardized core courses. The random assignment of students to professors, along with a vast amount of data on both professors and students, allows us to
examine how professor quality affects student achievement free from the usual problems of self-selection. Furthermore, performance in USAFA core courses is a consistent measure of student achievement
because faculty members teaching the same course use an identical syllabus and give the same exams during a common testing period. Finally, USAFA students are required to take and are randomly assigned
to numerous follow-on courses in mathematics, humanities, basic sciences, and engineering. Performance in these mandatory follow-on courses is arguably a more persistent measurement of student learning.
Thus, a distinct advantage of our data is that even if a student has a particularly poor introductory course professor, he or she still is required to take the follow-on related curriculum.”
Their methodology:
“We start by estimating professor quality using teacher value-added in the contemporaneous course. We then estimate value-added for subsequent classes that require the introductory course
as a prerequisite and examine how these two measures covary. That is, we estimate whether high- (low-) value-added professors in the introductory course are high- (low-) value-added professors for student
achievement in follow-on related curriculum. Finally, we examine how these two measures of professor value-added (contemporaneous and follow-on achievement) correlate with professor observable attributes
and student evaluations of professors. These analyses give us a unique opportunity to compare the relationship between value-added models (currently used to measure primary and secondary teacher quality) and
student evaluations (currently used to measure postsecondary teacher quality).
Their findings:
Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow-on course achievement. However,
our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank,
teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value-added but positively correlated with follow-on course value-added. Hence, students of less
experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow-on related curriculum.
For example:
As an illustration, the introductory calculus professor in our sample who ranks dead last in deep learning ranks sixth and seventh best in student evaluations and contemporaneous value-added, respectively.
Required reading for all serious teachers and students and Deans. Ungated version
8 comments
Comments feed for this article
June 11, 2010 at 9:26 pm
Anonymous
“Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy of this practice.”
The more relevant test is whether student evaluations are positively correlated with the propensity to give to the alumni fund 20 years later. I have vaguely heard (ha ha!) of such an institutional objective function, but I simply forget which institution this was…. 😉
June 12, 2010 at 10:31 pm
In Defense of Teacher Evaluations « Cheap Talk
[…] | by jeff Everyone is jumping on the bandwagon, including Tyler Cowen, Greg Mankiw, and even Sandeep. They are all trumpeting this study whose bottom line is that student evaluations of teachers are […]
June 13, 2010 at 2:21 am
gnossie
This is a great excuse for me to continue being a crappy teacher.
The way I read this study, if you’re determined to learn something, having a bad prof is almost a gift: you’re forced to get your bearings, figure it out yourself, develop habits of resourcefulness, initiative, etc. on your own, ’cause you know that if you don’t learn the stuff, you ain’t gonna be taught it.
Meanwhile, good profs inspire, highlight key points, and supply additional material. When this kind of support is missing in later classes, I think students get a bad attitude and their involvement goes down.
Thus, to do my students right in the long-term, I must be a cruddy teacher. The sooner they get disgusted with me and conclude that if they’re going to learn something, they’d damn well better see to it themselves, the better off they’ll be in the long run.
February 7, 2011 at 7:33 pm
Using student evaluations to measure teaching effectiveness « Real Learning Matters
[…] Harvard professor of economics Greg Mankiw, and Northwestern professor of managerial economics Sandeep Baliga. Michael Bishop, a contributor to Permutations (“official blog of the Mathematical Sociology […]
June 12, 2013 at 8:42 am
Intan
Andrew – Thanks Bruce! For the dance floor shots I was using a off camera flash and buicnong the ceiling with it and i had a white card to add some fill. It was high and dark in their, but i just powered up when I needed to.
June 12, 2013 at 10:30 pm
chdryy
c5edKo admtgiuvedas
September 4, 2013 at 12:00 pm
Oleinik
Here we go again. School can definitely crazy and it’ll be fun takinlg to you in the wee hours of the night when we’re crazily working on assignments. I know this quarter will go by fast & be chaotic because of my journalism classes & all the other stuff that needs to be done > those classes are always the best!Good luck ^^!![]
October 17, 2013 at 4:24 pm
Grazy
i envy good speakers~ they seem to catch your atotitnen and inspire you.it’s funny; buying a new laptop and installing these new programs sounds like a hassle, but i would love to be doing that! at the moment i’m just trialling photoshop and adobe indesign. it’s such a hassle ;__; i wish i could have loads of design programs []