Evaluating Teachers: What About Doing it the Old-Fashioned Way?

As part of our ongoing obsession with improving public education, we bring you a new study from Jonah E. Rockoff of Columbia Business School and Cecilia Speroni, a former doctoral student at Columbia’s Teachers College, that explores the power of objective and subjective teacher evaluations. While an emphasis on merit pay and test scores can lead to widespread cheating (as covered in this week’s Freakonomics Marketplace podcast), not to mention the occasional Matt Damon outburst, Rockoff and Speroni offer a potential glimmer of hope for the old-fashioned approach: the study finds that subjective teacher evaluations for New York City teachers had strong predictive power for future student performance. Here’s the abstract:

A substantial literature documents large variation in teacher effectiveness at raising student achievement, providing motivation to identify highly effective and ineffective teachers early in their careers. Using data from New York City public schools, we estimate whether subjective evaluations of teacher effectiveness have predictive power for the achievement gains made by teachers’ future students. We find that these subjective evaluations have substantial power, comparable with and complementary to objective measures of teacher effectiveness taken from a teacher’s first year in the classroom.

The authors used subjective evaluations from both applicant interviews for a certification program and mentors who worked with teachers their first year, as well as objective evaluations “based on student achievement data from their first year of teaching.”

Among the many knocks on the new push for objective evaluation measures is that they fail to capture the nuances of teaching, which the authors believe traditional subjective methods do much better.

This is an especially noteworthy finding, considering that variation in subjective evaluations likely also captures facets of teaching skill that may affect outcomes not captured by standardized tests.

But not so fast — the researchers noted two important things about teachers that received high evaluations from their mentors: their students had higher-than-average test scores prior to entering their classrooms, and they taught fewer minority or economically disadvantaged students. This could mean that better teachers go to schools with better students, or that a mentor’s impression of a teacher is influenced simply by their sample of students.

Rockoff and Speroni call for more research into the nuances and possibilities of subjective and objective testing, writing that “policymakers will need to have a better understanding of the power and limitations of the measures they use in establishing incentives and accountability for teachers.” Which seems to sum up the current debate quite well.


How good are parents at evaluating teachers, and if parents had school/teacher choice, what evaluation systems would school systems generate to attract parents and differentiate their product?

Would different centralized systems like Carfax form organically?


When I went to school I had some spectacularly bad teachers. A few good teachers. The rest were mediocre. I had one teacher who, on our first day, gave us a multiple choice test that had all the correct answers underlined. When this was pointed out, he lied and said that was just to throw us off. One math teacher who spent 8 months teaching us adding and subtracting fractions. I couldn't get out of his class, even though I was taking algebra the previous year.

Throw in schools where the teachers really didn't care what was going on in the classes or on the school grounds, if or when students attended, and pretty soon, I found myself less motivated then the worse teachers.

I know that things got cleaned up a year after I left. But still there are many students who are simply not motivated, or in fact are contra-motivated towards school and learning. Often because of family situations, lack of parenting, peer pressure, drugs and alcohol... and these problems are often worse in poor inner city schools as parents struggle without many or any supports, and as cycles of poverty and poor education are passed down the generations.

Out of all the talk of improving schools and teachers, I never hear how these issues are going to be addressed. Catch those cheating teachers.

But what about all the students who show up physically, but check out their minds as soon as they sit down? It can be hard to motivate a child, especially if your a single parent working several part time jobs, and your children are latch key kids.

You can believe it or not, but this is a big problem. In my area, (British Colombia, Canada) they have taken several approaches:

Pass, pass pass. Pass those kids, no matter what. Can't write? Can't read? Can't do math? Pass them anyways. I'm not sure how I feel about this. Is it helping, or hurting those students? Should it be done for those who don't have the ability to pass, and not the stoners or otherwise uninitiated?

Management have been replaced with people who are willing to do their job, and document problem teachers, then force retirement, or termination. I've worked union jobs, and in all cases the problem people were kept on the job because managers failed to document problems, making it much harder to fire problem employees. Eventually they got fed up, fired them, then complained when the union forced them to be rehired. At which point some become more careful, but continue to be problems. Managers who collect the paperwork and follow procedure don't have problems firing people. But it does take time. So does getting sued for wrongful dismissal.

Offer grade 12 and under, as well as other courses for free for any citizen who wants them, for free, through online, in person or a combination of the two course styles. When eventually the person realizes the need, or has the drive to better educate themselves, the ability is there, even if they had their shot. It also allows anyone to brush up on areas they haven't used since grade 11 or 12. Then they can get higher education easier and cheaper.


Rex McClure

Evaluations, particularly those supplied by students, can be quit misleading and may not reflect short-term outcomes (learning) or long-term outcomes (success in employment). I did a quick analysis of data from ratemyprofessor.com and found that teacher quality and easiness correlated strongly (r = 0.581, p < 0.001). About one thrid of the variance in my ratings are subsummed by something that is under my control--my ability to curve exam grades. As a member of homoeconomus, I know that my path to tenure goes through Lower-the-barville.

Somewhere, homoegalitaricus is shaking a fist in anger.


Is that such a bad thing? I found that I usually learned a lot more in my easier classes.


What about reducing summer vacation lengths. If I experience (as in time spent accomplishing a goal) truely does pay the dividends that many recent reports are saying wouldnt it follow that children spending more time in school earn more education.

It may be a tough sell since it seems to be ingrained philosophy.

Wikipedia asserts teachers worked about 35-46hrs per week, and I think this is not including vacation times (ie summer break).


As someone who is dating a teacher and knows a surprising amount of them I can tell you that from anecdotal evidence the summer is hugely damaging to a child's education. Teachers have to spend the first several weeks of school refreshing from the previous year. I can also tell you most teachers would quit before giving up summers.

I am not sure I believe the 35-46 hour a week number, or I would guess it is much closer to the 46. Teacher technically have to be at school for 40 hrs/week and almost all of the ones I know either work more during the week or spend a majority of one of the weekend days working as well. My girlfriend might be an overachiever but I would say she works closer to 55 hours a week as she typically works longer than me and I work 9 hour days, and she spends a good chunk of Sunday working as well. From my experience many teachers (the good ones at least) function like this.


Joshua Northey

I was in high school in the late 90s. The teachers were physically at the school from ~ 8-4:00. A few would stay longer, but the mean was definitely under 9 hours. They also had 1 or 2 hours of prep time a day.

This is at a poor public high school (albeit in a good education state). From what I saw then and know now (I was a teaching assistant for several college courses in a variety of fields) I doubt many of them did more than 1.5 hours of work a day outside of class.

So that is 165-170 9-10 hour days, add in another 2 weeks of 8 hour inservice days for another 80 hours.

That is still a good 300 hours (or 2 months) less work than most of the rest of the working population.

Some of my teachers were great, but most of them were just people from the bottom rung of college graduates, and several were downright horrible. Certainly my job is a lot more mentally demanding than being a teacher, also requires a college degree, and requires me to work 5 days a week year round, and some saturdays. Somehow I manage without summers off.

My grandfather (whom I love dearly) was a well regarded high school math teacher for 35 years and I can tell you from first hand experience that he was horrible at math, and a poor teacher (just told lots of stories about football and hockey). He did go in at 6:30 each morning and stay until 5:00. He was frequently the first there and one of the last to leave. I don't recall him doing any work outside of school at all.



Really? That's an interesting thought. What made them easier? Was it the course content or the way in which it was presented?

I can see that a teacher who is capable of breaking the learning into bite-size, chewable chunks will get farther with students than the teacher who stuffs bug uncut hunks down throats. However, in general, I found that easy courses were...not worth the time spent. That is, that I could have learned just as much reading a short book on the topic as sitting through a year or half year of too easy, too many small steps, not enough covered class.


Oops, didn't realize the first hadn't posted yet and ended up with it here, rather than above where it was intended!

Ex teacher

So you conclude that the major flaw in this report is it's failure to control for factors (students entering with high test scores already/fewer minority and disadvantaged students (arguably the crux of this issue surrounding good teaching because most people can teach already smart children. It's much more difficult to raise achievement in student who start behind the starting line)). It also presents a correlation rather than causation, for you point out that the subjective ratings from mentors may be influenced by their sample of students. This was not a very insightful post...

P.s. most teacher evaluation systems consider both data and professional evaluations in tandem.

P.p.s. I don't think Matt Damon's rant had ANYTHING to do with merit pay or testing. That just seemed like a desperate use of that video. In the video he is arguing against the concept of tenure creating a discincintive to teach well. I'd rather see you guys take up THAT issue. It would be much more insightful.



So now that we have used up all the talking points. Why don't we take a look at this from the logical point of view. In the first place teachers in the US know that if they are a part of a union they most likely won't get fired, so good teacher bad teacher they play on an even field. With no worries the children suffer wether in subburia or the inner city. Standardized testing hurts the school the teachers go to other schools in the area. The answer control the unions, or do away with the unions.

The second issue is parents should teach and demand that their children behave. Teachers are not baby sitters! They should teach the subjects they were traind to teach. Raise your kids!!! Things will get better.


I am sure some teachers do work 55hrs weeks, also sure some dont.

I am also quite sure that the whole summer vacation is needed for de-stressing is a myth. I have no real evidence but shooting from the hip I feel like it causes stress (an equal amount as having the entire summer off, cant say). I just know that teachers are all wound up prior to summer ending and for the first few weeks before "things settle down". Without that type of break, that type of stress would be eliminated. There is a similiar phenomenon towards the end of the year. I believe year long schooling would defeat a lot of these things.

I am not completely unreasonable berating teachers for not working, its tough duty. You have to deal with a whole population of people that are not adults an do not have the same concerns and worries you do, its hard to relate to individuals like that all the time. So just up and remove summer vacation. Just shorten the breaks particularly in summer. Without a doubt it would be better for the students academically. Maybe a shortened summer day, with a few classes? Not sure what the best solution is, but I do believe this is a large part of declining achievement.



So something that's historically been true is now the cause of "declining achievement"? I can think of reasons that might be so, I guess, but none of them has to do with the teachers or the schools. This would be a stronger point if you gave some reasons why this is *now* the cause.

Many teachers do end up working year-round (either teaching summer school or working at camps and other "summer" places) as well -- so they may not be that averse to the idea. However, I'm sure that taxpayers would be averse to an added 8 weeks of pay. In many areas of the Northeast and midwest, schools are not air-conditioned, either. Would that be included in the costs -- or would teachers be expected to teach hot, unhappy children in rooms that are often 85-95 degrees in my district in the summer (we know this from summer school).


Wow, what a shocker. I have a crazy idea. How about we evaluate teachers the way everyone at a corporate job (at least reasonably competent corporations) gets evaluate. Every year our boss (for teachers, the Principal) evaluates us. Yes, it's qualitative and subjective. And our raises, bonuses, promotions depend on that evaluation AND metrics such as company revenue and profitability (for teachers, let's say test scores).

OK, can we finally end this silly discussion about teacher evaluations? Am I missing something? Is it more complicated than I'm making it out to be? Yes, there are bad bosses, poorly designed performance reviews in companies, etc., but there are also good bosses and well designed evaluation systems. Let's hold teachers accountable and their Principals just like good corporations do.

Eric M. Jones.

Ah yes....memories of youth. Stabbings on the grade 1-6 playground were fairly rare. My fifth grade teacher Miss DePuy beat a little girl unconscious by slamming her dead against the blackboard. Her limp body was delivered to the school nurse. Soon the girl's mother and aunt showed up at the school with baseball bats to address the issue with Miss DePuy. The janitor had to hide her in the coal cellar until they left.

You can't really blame the teacher. Sometimes her lumbago would flare up....and we were always on guard.

The school had 1/3 whitebread kids like me, 1/3 tough poor Irish and black kids from "The Patch" (who had a continuous gang war) and 1/3 kids from the reform school across the highway. The school served as a convenient "dumping ground" for teachers who had gotten into difficulties at other schools.

It seemed fitting that we would be marched like prisoners to the auditorium to see the early space shot missiles explode on takeoff. That's how my life seemed.



Education 'experts' have a difficult time releasing control of the notion that the general public -- parents in particular -- cannot effectively evaluate teacher performance.

It's a slippery comparison. These 'experts' usually point out that parent evaluation is not perfect, without also pointing out that expert or formulaic accountability isn't either. The question isn't whether it is perfect, it's which one is better.

I trust parents more than experts. I agree with what Gov. Christie said here (http://ourdinnertable.wordpress.com/2011/04/07/gov-christie-on-how-to-tell-a-good-teacher-from-a-bad-teacher/)


I know this posting is older now, but here's a link to an evaluation rubric being used in Tennessee: http://www.team-tn.org/assets/educator-resources/TEAM_Educator_Rubric.pdf (non-validated, btw) A quick glance over it makes it look, you know, like something pretty basic and obvious. But, really start looking at some of the expectations and the differences between them.

For instance to get a 5 rating a teacher must (among other things in this one category):

"overlook inconsequential behavior"
"deal with students who have caused disruptions rather than the entire class"
"attend to disruptions quickly and firmly."

A 3 level ("at expectation" -- but not good enough to receive tenure) states:

"students are mostly well-behaved and on task, some minor learning disruptions may occur"
" teacher overlooks some inconsequential behavior, but other times addresses it, stopping the lesson."
"teacher deals with students who have caused disruptions, yet sometimes he or she addresses the entire class."

Those all sound good, right? I have a picture in my head of what they're driving at there.

But clearly, there can be a huge difference of opinion on inconsequential behavior (and truly, it depends on the class and the specific students, right?), especially between a teacher who has been in a room with the students for weeks and an observer there for an evaluation, which is usually 20-45 minutes maximum.

What is an inconsequential behavior for one child (say, elbow bumps pencil off table) might be a huge warning sign that needs to be addressed for another child. A teacher would know this, but an observer would not and might ding the teacher for stopping a lesson to quickly ask the child to pick it up or to pick it up and move seats. The teacher would know, but couldn't prove that without that "disruption" the child would have quickly escalated to far more "consequential" behaviors. The rater might rate this as wrong on several levels, not realizing that without it, there would have been fireworks. That's a bind for any teacher being observed.

Also, if it is a group lesson, how is a teacher to address a student individually, not the whole class, while also not interrupting the lesson of the whole class?

These sorts of rubrics sound great -- though I can't believe that there is a lot of reliability between raters -- nor any time or effort put in toward that goal. So, it comes down to does your principal like you as a teacher and think you're doing a good job, does the principal respect your style, even if it's different than his/her own, etc -- and how is that different than it ever was?

Well, I guess it's different in that there are offices full of people at the state level drawing salaries to come up with these rubrics.


Ms. H

After reading these comments, it still amazes me how much people do NOT know about what goes on in a classroom or school. Yet there are so many opinions that are usually derived from personal experiences and applied to all. As in every field, there will be hard workers and less dedicated workers (educators and corporate employees). I can say, as an educator, that far more of us are hard-workers giving many more hours than the 7-3p school day. Who's to say the teacher leaving right at 3p isn't off to a professional development meeting to improve his effectivness in the classroom, or to a graduate class to expand her knowledge on teaching practices. Who's to say they aren't headed to an after-school tutoring program, or to coach on of the many extra-curricular activities. Finally, please do not assume that teachers do not support eliminating summer breaks. Most of us want the break shortened or eliminated because retention would increase in students. In ohio, it is BUSINESS that does not support that notion because of the loss in revenue, not educators.

I apologize for all those that have had such a bad educational experience. It was not fair to you nor is it fair to any student that has to come through a system as some of you have described. I was blessed to have some exceptional teachers who in turn inspired me to become an educator. I continue to learn from them as I develop my teaching practices because now I call them my colleagues.


Samantha Ebeling

In this article, it seems that subjective teaching proves to be more succesful in really helping students. Although objective teaching shows the teachers who focus more on test grades and the ones who have classes with higher test grades, many times those scores are the outcome of cheating. Teachers who focus on all around teaching, rather than test scores are more likely to have better, more well-rounded students. Also, some teachers just have smarter, more privleged students than those who teach at schools with economically disadvantaged ones. This article does not really determine which type of teaching is more useful and helpful.

steve jasinowski

I belive that teachers who only care nad pay attention to the grades that the students get on a test or homework assignment are the teachers that their students have a greater cheating percentage. this is argueable because when a teacher does not follow up with corrections of anything it would make kids look for the fastest and easiest way of getting it done without the quality.