Evaluating Teachers: What About Doing it the Old-Fashioned Way?

As part of our ongoing obsession with improving public education, we bring you a new study from Jonah E. Rockoff of Columbia Business School and Cecilia Speroni, a former doctoral student at Columbia’s Teachers College, that explores the power of objective and subjective teacher evaluations. While an emphasis on merit pay and test scores can lead to widespread cheating (as covered in this week’s Freakonomics Marketplace podcast), not to mention the occasional Matt Damon outburst, Rockoff and Speroni offer a potential glimmer of hope for the old-fashioned approach: the study finds that subjective teacher evaluations for New York City teachers had strong predictive power for future student performance. Here’s the abstract:

A substantial literature documents large variation in teacher effectiveness at raising student achievement, providing motivation to identify highly effective and ineffective teachers early in their careers. Using data from New York City public schools, we estimate whether subjective evaluations of teacher effectiveness have predictive power for the achievement gains made by teachers’ future students. We find that these subjective evaluations have substantial power, comparable with and complementary to objective measures of teacher effectiveness taken from a teacher’s first year in the classroom.

The authors used subjective evaluations from both applicant interviews for a certification program and mentors who worked with teachers their first year, as well as objective evaluations “based on student achievement data from their first year of teaching.”

Among the many knocks on the new push for objective evaluation measures is that they fail to capture the nuances of teaching, which the authors believe traditional subjective methods do much better.

This is an especially noteworthy finding, considering that variation in subjective evaluations likely also captures facets of teaching skill that may affect outcomes not captured by standardized tests.

But not so fast — the researchers noted two important things about teachers that received high evaluations from their mentors: their students had higher-than-average test scores prior to entering their classrooms, and they taught fewer minority or economically disadvantaged students. This could mean that better teachers go to schools with better students, or that a mentor’s impression of a teacher is influenced simply by their sample of students.

Rockoff and Speroni call for more research into the nuances and possibilities of subjective and objective testing, writing that “policymakers will need to have a better understanding of the power and limitations of the measures they use in establishing incentives and accountability for teachers.” Which seems to sum up the current debate quite well.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.



View All Comments »
  1. Joshua says:

    How good are parents at evaluating teachers, and if parents had school/teacher choice, what evaluation systems would school systems generate to attract parents and differentiate their product?

    Would different centralized systems like Carfax form organically?

    Thumb up 2 Thumb down 0
  2. Mike says:

    When I went to school I had some spectacularly bad teachers. A few good teachers. The rest were mediocre. I had one teacher who, on our first day, gave us a multiple choice test that had all the correct answers underlined. When this was pointed out, he lied and said that was just to throw us off. One math teacher who spent 8 months teaching us adding and subtracting fractions. I couldn’t get out of his class, even though I was taking algebra the previous year.

    Throw in schools where the teachers really didn’t care what was going on in the classes or on the school grounds, if or when students attended, and pretty soon, I found myself less motivated then the worse teachers.

    I know that things got cleaned up a year after I left. But still there are many students who are simply not motivated, or in fact are contra-motivated towards school and learning. Often because of family situations, lack of parenting, peer pressure, drugs and alcohol… and these problems are often worse in poor inner city schools as parents struggle without many or any supports, and as cycles of poverty and poor education are passed down the generations.

    Out of all the talk of improving schools and teachers, I never hear how these issues are going to be addressed. Catch those cheating teachers.

    But what about all the students who show up physically, but check out their minds as soon as they sit down? It can be hard to motivate a child, especially if your a single parent working several part time jobs, and your children are latch key kids.

    You can believe it or not, but this is a big problem. In my area, (British Colombia, Canada) they have taken several approaches:

    Pass, pass pass. Pass those kids, no matter what. Can’t write? Can’t read? Can’t do math? Pass them anyways. I’m not sure how I feel about this. Is it helping, or hurting those students? Should it be done for those who don’t have the ability to pass, and not the stoners or otherwise uninitiated?

    Management have been replaced with people who are willing to do their job, and document problem teachers, then force retirement, or termination. I’ve worked union jobs, and in all cases the problem people were kept on the job because managers failed to document problems, making it much harder to fire problem employees. Eventually they got fed up, fired them, then complained when the union forced them to be rehired. At which point some become more careful, but continue to be problems. Managers who collect the paperwork and follow procedure don’t have problems firing people. But it does take time. So does getting sued for wrongful dismissal.

    Offer grade 12 and under, as well as other courses for free for any citizen who wants them, for free, through online, in person or a combination of the two course styles. When eventually the person realizes the need, or has the drive to better educate themselves, the ability is there, even if they had their shot. It also allows anyone to brush up on areas they haven’t used since grade 11 or 12. Then they can get higher education easier and cheaper.

    Thumb up 3 Thumb down 3
  3. Rex McClure says:

    Evaluations, particularly those supplied by students, can be quit misleading and may not reflect short-term outcomes (learning) or long-term outcomes (success in employment). I did a quick analysis of data from ratemyprofessor.com and found that teacher quality and easiness correlated strongly (r = 0.581, p < 0.001). About one thrid of the variance in my ratings are subsummed by something that is under my control–my ability to curve exam grades. As a member of homoeconomus, I know that my path to tenure goes through Lower-the-barville.

    Somewhere, homoegalitaricus is shaking a fist in anger.

    Well-loved. Like or Dislike: Thumb up 6 Thumb down 0
  4. Jrod says:

    What about reducing summer vacation lengths. If I experience (as in time spent accomplishing a goal) truely does pay the dividends that many recent reports are saying wouldnt it follow that children spending more time in school earn more education.

    It may be a tough sell since it seems to be ingrained philosophy.

    Wikipedia asserts teachers worked about 35-46hrs per week, and I think this is not including vacation times (ie summer break).

    Thumb up 2 Thumb down 0
    • Bob says:

      As someone who is dating a teacher and knows a surprising amount of them I can tell you that from anecdotal evidence the summer is hugely damaging to a child’s education. Teachers have to spend the first several weeks of school refreshing from the previous year. I can also tell you most teachers would quit before giving up summers.

      I am not sure I believe the 35-46 hour a week number, or I would guess it is much closer to the 46. Teacher technically have to be at school for 40 hrs/week and almost all of the ones I know either work more during the week or spend a majority of one of the weekend days working as well. My girlfriend might be an overachiever but I would say she works closer to 55 hours a week as she typically works longer than me and I work 9 hour days, and she spends a good chunk of Sunday working as well. From my experience many teachers (the good ones at least) function like this.

      Thumb up 4 Thumb down 0
      • Joshua Northey says:

        I was in high school in the late 90s. The teachers were physically at the school from ~ 8-4:00. A few would stay longer, but the mean was definitely under 9 hours. They also had 1 or 2 hours of prep time a day.

        This is at a poor public high school (albeit in a good education state). From what I saw then and know now (I was a teaching assistant for several college courses in a variety of fields) I doubt many of them did more than 1.5 hours of work a day outside of class.

        So that is 165-170 9-10 hour days, add in another 2 weeks of 8 hour inservice days for another 80 hours.

        That is still a good 300 hours (or 2 months) less work than most of the rest of the working population.

        Some of my teachers were great, but most of them were just people from the bottom rung of college graduates, and several were downright horrible. Certainly my job is a lot more mentally demanding than being a teacher, also requires a college degree, and requires me to work 5 days a week year round, and some saturdays. Somehow I manage without summers off.

        My grandfather (whom I love dearly) was a well regarded high school math teacher for 35 years and I can tell you from first hand experience that he was horrible at math, and a poor teacher (just told lots of stories about football and hockey). He did go in at 6:30 each morning and stay until 5:00. He was frequently the first there and one of the last to leave. I don’t recall him doing any work outside of school at all.

        Thumb up 1 Thumb down 5
      • Jen says:

        Really? That’s an interesting thought. What made them easier? Was it the course content or the way in which it was presented?

        I can see that a teacher who is capable of breaking the learning into bite-size, chewable chunks will get farther with students than the teacher who stuffs bug uncut hunks down throats. However, in general, I found that easy courses were…not worth the time spent. That is, that I could have learned just as much reading a short book on the topic as sitting through a year or half year of too easy, too many small steps, not enough covered class.

        Thumb up 3 Thumb down 0
      • Jen says:

        Oops, didn’t realize the first hadn’t posted yet and ended up with it here, rather than above where it was intended!

        Thumb up 0 Thumb down 0
  5. Ex teacher says:

    So you conclude that the major flaw in this report is it’s failure to control for factors (students entering with high test scores already/fewer minority and disadvantaged students (arguably the crux of this issue surrounding good teaching because most people can teach already smart children. It’s much more difficult to raise achievement in student who start behind the starting line)). It also presents a correlation rather than causation, for you point out that the subjective ratings from mentors may be influenced by their sample of students. This was not a very insightful post…

    P.s. most teacher evaluation systems consider both data and professional evaluations in tandem.

    P.p.s. I don’t think Matt Damon’s rant had ANYTHING to do with merit pay or testing. That just seemed like a desperate use of that video. In the video he is arguing against the concept of tenure creating a discincintive to teach well. I’d rather see you guys take up THAT issue. It would be much more insightful.

    Thumb up 4 Thumb down 2
  6. Joe says:

    So now that we have used up all the talking points. Why don’t we take a look at this from the logical point of view. In the first place teachers in the US know that if they are a part of a union they most likely won’t get fired, so good teacher bad teacher they play on an even field. With no worries the children suffer wether in subburia or the inner city. Standardized testing hurts the school the teachers go to other schools in the area. The answer control the unions, or do away with the unions.

    The second issue is parents should teach and demand that their children behave. Teachers are not baby sitters! They should teach the subjects they were traind to teach. Raise your kids!!! Things will get better.

    Thumb up 0 Thumb down 1
    • Jrod says:

      I am sure some teachers do work 55hrs weeks, also sure some dont.

      I am also quite sure that the whole summer vacation is needed for de-stressing is a myth. I have no real evidence but shooting from the hip I feel like it causes stress (an equal amount as having the entire summer off, cant say). I just know that teachers are all wound up prior to summer ending and for the first few weeks before “things settle down”. Without that type of break, that type of stress would be eliminated. There is a similiar phenomenon towards the end of the year. I believe year long schooling would defeat a lot of these things.

      I am not completely unreasonable berating teachers for not working, its tough duty. You have to deal with a whole population of people that are not adults an do not have the same concerns and worries you do, its hard to relate to individuals like that all the time. So just up and remove summer vacation. Just shorten the breaks particularly in summer. Without a doubt it would be better for the students academically. Maybe a shortened summer day, with a few classes? Not sure what the best solution is, but I do believe this is a large part of declining achievement.

      Thumb up 2 Thumb down 0
      • Jen says:

        So something that’s historically been true is now the cause of “declining achievement”? I can think of reasons that might be so, I guess, but none of them has to do with the teachers or the schools. This would be a stronger point if you gave some reasons why this is *now* the cause.

        Many teachers do end up working year-round (either teaching summer school or working at camps and other “summer” places) as well — so they may not be that averse to the idea. However, I’m sure that taxpayers would be averse to an added 8 weeks of pay. In many areas of the Northeast and midwest, schools are not air-conditioned, either. Would that be included in the costs — or would teachers be expected to teach hot, unhappy children in rooms that are often 85-95 degrees in my district in the summer (we know this from summer school).

        Thumb up 1 Thumb down 2
  7. Ben says:

    Wow, what a shocker. I have a crazy idea. How about we evaluate teachers the way everyone at a corporate job (at least reasonably competent corporations) gets evaluate. Every year our boss (for teachers, the Principal) evaluates us. Yes, it’s qualitative and subjective. And our raises, bonuses, promotions depend on that evaluation AND metrics such as company revenue and profitability (for teachers, let’s say test scores).

    OK, can we finally end this silly discussion about teacher evaluations? Am I missing something? Is it more complicated than I’m making it out to be? Yes, there are bad bosses, poorly designed performance reviews in companies, etc., but there are also good bosses and well designed evaluation systems. Let’s hold teachers accountable and their Principals just like good corporations do.

    Thumb up 2 Thumb down 1
  8. Eric M. Jones. says:

    Ah yes….memories of youth. Stabbings on the grade 1-6 playground were fairly rare. My fifth grade teacher Miss DePuy beat a little girl unconscious by slamming her dead against the blackboard. Her limp body was delivered to the school nurse. Soon the girl’s mother and aunt showed up at the school with baseball bats to address the issue with Miss DePuy. The janitor had to hide her in the coal cellar until they left.

    You can’t really blame the teacher. Sometimes her lumbago would flare up….and we were always on guard.

    The school had 1/3 whitebread kids like me, 1/3 tough poor Irish and black kids from “The Patch” (who had a continuous gang war) and 1/3 kids from the reform school across the highway. The school served as a convenient “dumping ground” for teachers who had gotten into difficulties at other schools.

    It seemed fitting that we would be marched like prisoners to the auditorium to see the early space shot missiles explode on takeoff. That’s how my life seemed.

    Thumb up 0 Thumb down 1