Skeptical Approaches to Student Evaluations of Faculty

The college where I dispense wisdom is very small, with fewer than 1000 students. We are a weird little school in a weird, isolated little town, surrounded by much bigger players who suck up most of the (decimated) higher ed budget and leave us with the scraps. Thus, we have to sell ourselves as best we can, by focusing on our intimacy, meaning issues like student evaluations really matter. We have spent many fraught hours haggling over paper evals vs. online; the exact wording of questions; how responses can be quantified; and to what extent student evaluations should matter for tenure and promotion.

When I sat down to research and compose a post about student evaluations of faculty (SEF), I did not expect to be fascinated by my findings, in part because of the often annoying baggage of it all. But you know what? There’s some really intriguing research out there on this topic. Here is a decent summary of major concerns with a useful preliminary bibliography that I encourage you to read in its entirety. One item caught my fancy, though, because it hits that sweet spot we love so much here at School of Doubt, the overlap of education and skepticism: It’s called the Dr. Fox Effect.

From the Wikipedia entry:

In a critique of student evaluations of teaching, professor of law Deborah Merritt summarized the Dr. Fox Effect as it was observed in the first experiments, in which an actor gave a lecture to a group of ten under the guise of “Dr. Myron L. Fox”: “The experimenters created a meaningless lecture on ‘Mathematical Game Theory as Applied to Physician Education,’ and coached the actor to deliver it ‘with an excessive use of double talk, neologisms, non sequiturs, and contradictory statements.’ At the same time, the researchers encouraged the actor to adopt a lively demeanor, convey warmth toward his audience, and intersperse his nonsensical comments with humor. … The actor fooled not just one, but three separate audiences of professional and graduate students. Despite the emptiness of his lecture, fifty-five psychiatrists, psychologists, educators, graduate students, and other professionals produced evaluations of Dr. Fox that were overwhelmingly positive. … The disturbing feature of the Dr. Fox study, as the experimenters noted, is that Fox’s nonverbal behaviors so completely masked a meaningless, jargon-filled, and confused presentation.”

How freaking cool is that? Want something even cooler? Here’s footage from 1970 of the actor, Michael Fox, delivering the famous lecture:

It is unsurprising to discover that students (and people in general) respond positively to charismatic presenters. There are follow up studies using similar methods that delve into ideas of educational effectiveness, such as the difference between *believing* you have learned something because you enjoyed the presentation and *actually* learning something. Tricky stuff. Here, though, I just want to talk about the problem (is it a problem?) of educator charisma in the context of faculty evaluations. Do SEFs measure instructor likability above all else? Does being likable make one a more effective teacher?

This is a more complicated question than some might realize. For instance, we labor under a pervasive myth in our culture that the most successful teachers “make learning fun.” How many times have you heard this claim bandied about as objective unimpeachable truth? Have you ever wondered if it deserves that privileged status? I have, in part because I regularly do things to my students that I know perfectly well are not at all “fun.” So what do we mean when we say this? That the assignments themselves are fun or that content is being delivered in a fun way? You might think this is a concern only at the grade school level, but even at my level, college undergraduates, my course evaluations tend to focus on my personality rather than my course content. Not exclusively, but noticeably. I get comments every term, for example, identifying me as “funny.” Don’t get me wrong–I am in fact fucking hilarious–but why is that relevant to note on a document evaluating my teaching effectiveness? I know these questions seem to have intuitive answers, but when I really try to answer them I’m not sure they do. I think they are worth interrogating.

The most obvious potential problem revealed by the Dr. Fox experiment may be how easy it is to use charisma as a shield that hides incompetence. Unfortunately, the teaching profession can attract egotists who consider themselves entitled to a captive audience but suffer no desire to work for it. I have known some of these people; I bet you have as well. They exult in the adoration their ebullience inspires in impressionable youths; in extreme versions they exploit it for sex but mostly they just feed off the ego boost. Here’s a question: If those people are also delivering acceptable levels of content knowledge, is it ethically viable for them to seek this reciprocal ego stroke? Is it even possible to avoid it? I won our school’s award for superior teaching last year, supported by students. I can’t tell you how thrilled I was. Did I win because I’m funny? I hope not, but how would I know?