How to Game a Grading Curve

Students in three of Professor Peter Fröhlich‘s computer programming classes at Johns Hopkins University recently devised a method to game their final grades.  Frolich grades exams on a curve — the highest grade in the class, whatever it may be, becomes 100 percent, and “everybody else gets a percentage relative to it.”  So students collectively planned a boycott:

Because they all did, a zero was the highest score in each of the three classes, which, by the rules of Fröhlich’s curve, meant every student received an A.

“The students refused to come into the room and take the exam, so we sat there for a while: me on the inside, they on the outside,” Fröhlich said. “After about 20-30 minutes I would give up…. Then we all left.” The students waited outside the rooms to make sure that others honored the boycott, and were poised to go in if someone had. No one did, though.

Catherine Rampell discusses the strategy:

This is an amazing game theory outcome, and not one that economists would likely predict…

In this one-off final exam, there are at least two Bayesian Nash equilibria (a stable outcome, where no student has an incentive to change his strategy after considering the other students’ strategies). Equilibrium #1 is that no one takes the test, and equilibrium #2 is that everyone takes the test. Both equilibria depend on what all the students believe their peers will do.

If all students believe that everyone will boycott with 100 percent certainty, then everyone should boycott (#1). But if anyone suspects that even one person will break the boycott, then at least someone will break the boycott, and everyone else will update their choices and decide to take the exam (#2).

The problem is that Nash equilibrium theory alone doesn’t tell us what the students are more likely to do. Economists would say that the first equilibrium, where no one takes the exam, is unlikely to result because it is not “trembling hand perfect,” an idea that helped win Reinhard Selten win the Nobel Memorial Prize in Economics.

Fröhlich was impressed by the students’ scheme. “The students learned that by coming together, they can achieve something that individually they could never have done,” he wrote in an e-mail. “At a school that is known (perhaps unjustly) for competitiveness I didn’t expect that reaching such an agreement was possible.”  He has, however, revised his grading policy to prevent future gaming.

(HT: Sarah Martin)


tung bo

The students are also betting that their cohesiveness and game theoretical thinking will impress Prof. Frohlich. They stood to lose if the Professor took a legalistic approach: since none of the student participated in the exam by going into the room, the Professor can treat that as a forfeit. That means no grades for any student or failling grades for forfeits.

This risk should force some and then most student to defect given a large enough class. With a small class, it was possible to enforce nonparticipation.
Yet, the Professor can also offer a make-up take home test to replace the in class test. Without the public mutual monitoring, almost certainly some will defect.

Ultimately, the students were betting that Prof. Frolich was a 'nice' person and would not follow these other alternatives.

Matt

In high school I had a math teacher who had a similar grading curve policy for the final. A bunch of hands went up after she finished explaining it, at which point she amended it by saying that if everyone did very poorly (e.g., scored 0s) she would nullify the test and make us take a new one (and half the hands went down).

It all comes down to how much one thinks the teacher will appreciate the effort.

Jaime

It seems to me that this result was in all likelihood triggered by some previous action (on behalf of the teachers, university, etc) that removed any incentive for competition in the class as this is one of two mechanisms I see that would lead to this equilibrium. Alternatively it is possible some group of students with little chance of getting a high grade coerced their peers to take part in the plan.

This latter mechanism is less likely and not stable, as students actually have an incentive (that increases as more students are coerced into getting a 0%) to stand up to those that want to attempt the 0% grade strategy as the resisting student's grade is bound to see a greater difference compared to those other students in their class and hence end up with a more outstanding degree than his/her peers.

The former mechanism could be, for example, that the given class would not count for the final grade of the course/degree but a pass was still required to successfully complete the course/grade. This situation creates a stable equilibrium where all students are highly incentived to carry out such a strategy as there is no comparable gain between the "0%, 0 effort strategy" and "each student individually studying to get a passing grade" strategy, while there is a clear cost difference (in effort) that pushes the group towards the consensus arrived at in the anecdote.

Would be nice to get some background on this.

Read more...

J

Basically none of these equilibrium theories apply because we all knew what the other students were going to do beforehand, and on the day of the test.

I was in one of his classes, and I got 100 on my final using this strategy. First of all, I'm pretty sure Peter mentioned this possibility at the beginning of the year after he told us how his scale worked. If he hadn't I don't think anyone would have believed it would actually worked.

The reason we could organize this is that the class was using an online discussion website thing where everyone in the class was registered, and you could ask questions about homework or whatever and get an answer from your fellow students or the professor. So it was pretty easy for someone to post the idea and see if anyone was opposed to it. Most people in the class were stressed about finals and were happy to have a guaranteed 100 regardless of how they were doing in the class at the time, so they agreed to the scheme. A few other people didn't really want to do it but were willing to go along with it to help out their peers and avoid forcing a bunch of people to take the test. Then there was one kid who refused to skip the test because of moral and philosophical reasons. Somehow someone eventually convinced him to go in, take the test, but not hand it in, while everyone else stood outside and watched. It was pretty weird and I felt pretty bad for putting him in such an uncomfortable position, but he didn't turn it in so we all got the 100.

Read more...

Joe

Should they have all gotten 100%? I think that they should have gotten an undefined 0/0 if that were the maximum grade.

salviati

That is a loophole wide enough to drive a truckload of F's through. I love it. All the professor need do is to change the syllabus to read that he will "determine their percentage grade based on the ratio out of the highest score" and is thus free to determine that 0/0 = 0%

Asaf

0/0 is undefined... Not something a computer scientist would do :)

Ana Bee

One other thing to take under consideration the "dude, don't be an asshole" cause, which is sufficient to persuade a person not to break contract, when uttered near the classroom door.

Travis Idol

Exactly. Game theory in its simplest form assumes the participants don't know or can't directly influence the actions of the other players. In fact, the classic prisoner's dilemma works precisely because the defendants are kept separate. A boycott (think union strike) works because the participants stand around the entrance and attempt to discourage anyone thinking of crossing the picket line.

Lou

A more sophisticated curve would raise the median or mean to a desired value (say 75 or 80%). Thus everyone would get the same lower grade and remove the incentive for the top students to participate.

Peter

Lou, I was going to do the same thing. If the curve were based on the median score there should be, by definition, 50% of the class than can expect to receive a better grade. This would restore the separating equilibrium, by ensuring one student has an incentive to defect.

Khurram Makhdumi

If nobody entered the exam hall then they don't get zeroes, they were absent. To get zero, one must enter the exam hall and at least write name on the answer sheet. Given every student has an individual answer sheet, observing the divergent attitude directly is harder, thus the incentive to diverge and the more persistent equilibrium, i.e. everybody takes an exam.

jono gabono

Not really, on my University there is no such grading as "absent", if you where absent you get a big fat zero.

Enter your name...

The students took a significant risk, because the instructor had another option: to refuse to give them grades and instead assign them "incompletes".

Tom Fox

It seems to me like they all should have received an incomplete rather than a zero. Did he take attendance?

If this was my class trying to "game" my system I would have gamed them right back and said that there is a difference between not taking and not passing the exam. That would force them to all fill in "A" or something like that. Then we would see how much they trust each other.

J

The only reason this worked is the professor basically told us this was an option when he told us how his grading scheme worked. He doesn't think tests are a good way of evaluating performance so he was ok giving us all 100s, especially since the final wasn't worth a huge percentage of our final grade. And I'm pretty sure the only reason he changed his policy was because the rest of the CS department was furious at him.

Nate Vack

In my high school US History class, we actually did this same trick on our last weekly test. We all took the test, but had everyone score 0% by choosing "E" on an "A-D" multiple choice test. Though we had no way to detect defectors, everyone in our 25-person class cooperated. It was pretty great.

However: this was only one of many, many tests and we fully expected to be caught and made to re-take the test (which is exactly what happened), so the stakes were significantly lower for us.

Joe

A more interesting scenario would have been if the test was take home. Students would not be able to know if others were, in fact, staying true to their agreement. By standing outside of the classroom united, this uncertainty goes away.

Dave

That was their grade on the final but was it their final grade? Assuming the Professor issues final grades on the same type of curve based on other course work, midterms, etc. By getting identical grades on the final exam, all the students accomplished was to remove the final exam from the equation.

Andrew Kelly

Hello Freakonomics, it's the student from Rampell's article.

I wanted to clear up a few things about what really went down before things get out of hand in the comments section, like they have on many other websites as the story itself gets further from the truth.

I was in the Introduction to Programming course, which had students from seniors to freshmen, across many engineering and science majors. I'm a MechE senior who wanted to have a little programming under my belt before beginning my Masters program at Hopkins, and Python seemed like an interesting language.

Although an introductory class, the assignments were challenging and required a good chunk of time weekly. There was a midterm with a pretty mediocre average. It was only until the class of the year that we found about this 'boycott' potential. Fröhlich was holding a Q&A, and a classmate asked if it was true that the Intermediate Programming course did not take their midterm. Fröhlich explained that if no one took the exam, we would all receive A's 'based on a lousy Python script' he uses for his grading. So instead of some scheming, lazy students who were bent on gaming the system and the curve, we were merely handed an interesting to a 40 minute exam. Also, since there wasn't certainty that absolutely no student was going to walk in those doors meant we all studied for the exam. Did I study as efficiently as I would have without this looming over my head? No, but it did open up more time to study for my other exams and the lengths I went to organize the 'boycott' broke up the monotony of my 7th exam period at Hopkins. We used a public Google Drive spreadsheet to actively make sure everyone was on board, we reached out to those that weren't posting on our course website to see if they were still enrolled in the course and knew the terms of the final. Other classes were not as fortunate, but we were able to get 100% committal to the boycott from the students we knew were still in the class.

So I never really thought about it as a "curve" until these articles started putting it in that light. The final itself was only 10-15% of the final grade, so really your performance in those 10 assignments, midterm, group project, and participation defined your course grade.

As a senior in engineering at Hopkins, I've seem some crazy competitive behavior (but not like stereotypical pre-med behavior). It was extremely refreshing to hang out with a wide cross section of students outside the exam room the morning of that exam, having donuts and enjoying the whole situation. We pulled off someone I didn't think was possible, but it put more happy faces on Johns Hopkins students than I've ever seen as the result of a final exam.

tl;dr
We didn't set out to 'game the system'. Our class was told by Fröhlich on the last lecture of the semester that if no one took the exam, we would get an A. If one person took it, game on. We made sure everyone was on board, studied anyways, went outside the lecture hall on the day of the exam, had donuts, went home.

Read more...

PricklyPete

This would be much more interesting if the students did this without waiting outside the classroom ready to go inside in case of any defections.

SPENCER

Except they should have all gotten an incomplete for not taking the test. Hard to give a grade on a test never taken. More plausible, im assuming, would be if everyone wrote their name on the test and turned it in for a zero. Although zero, especially in a computer programming where zero divided by zero to yield a percentage should result in a logic error... hmmm

Aries

The common game theory example of prisoners dilemma, usually involve a situation where the participants do not know the other person action/testimony. In the case here, the participants have an almost perfect information of all the participants and a whole semester to work out and discuss the system. They also seem tohave a. relatively high degree of confidence in the teacher. predicted behavior. There seems to be little impediment to this plan and obviously they seem to have all be prepared for plan B. If anyone defect. Seems to me they are playing the game of mutually assured destruction deterrence to the fullest.

anonymous

the teacher had already said that if nobody took the test, he would give them all an A for the test. judging by that, I suspect the teacher wanted to see if they could actually do it.

Kreyg

We did this once for the first year exams in grad school (Econ PhD). After studying the entire summer, a large group of us were in the office of the Macro professor who asked us if we looked at any of the models that dealt with money from a previous professor. We said of course not, that was not what we learned in class.

Clearly, he was going to ask about one of these models and none of us had ever seen one. The exams are curved so we all agreed to not look at them so that no one had to freak out and try to learn an entire set of models within a week.

I'll never forget how much the one girl bragged about getting the scholarship for achieving the best score on the macro exam after she stabbed us all in the back and looked up the money models. I could say I don't get satisfaction when I see that she took seven years to graduate and did not obtain a permanent academic position...but I'd be lying. Schadenfreude for the win.

Read more...

Garam

I wonder, if I had done this in my experimental econ class, would we have deserved an A+ for putting our knowledge of game theory into action?

gpadugan@ymail.com

this makes no sense. the professor is still held accountable to the rules of the school. so the school could (and should over rule). besides, if we go by the numbers (which is what the students did) then go all the way. 0% has no value. It can not be 100% and 0% so technically speaking the test is null it and should be required to be re taken. you can say everyone passed with 100 or that everyone failed with 0. The professor could say that they all get a 0 instead of 100. he's just doing this for the press. I'm an adjunct professor and a sysllubus is not a contract, not a legal document, it's a guide. to take it literal as the students did is quite alarming indeed. they shall have plenty of time to continue this line of thinking in the occupy camps when they can't get a job.

Et

You are mistaken. The professors devise the grading mechanics, not the school (within reason). This was a flaw present in Professor Fröhlich‘s syllabus and was made apparent in the syllabus reviewed at the start of every semester. I've taken a number of his courses and it was discussed semi-jokingly a number of times in the past. Nobody had actually taken advantage of it before, perhaps in part because of the difficulty of guaranteeing collaboration of all students in previous years.

The key difference this year was that the class started using a discussion board (Piazza, specifically) to ask and answer questions related to course material throughout the semester, and participation was mandatory (graded), so all students in the course were active on the discussion board to some degree. It was on this board, as well as a google document, that the class organized (and debated, quite lengthily) the plan to not take the final.

The professor was aware of this discussion, but could not simply change his grading policies mid-semester. It was not until after the semester ended that he was able to modify the policy. Thus, the plan went through and was successful on several occasions in 3 different courses throughout the semester.

Source: I was in one of the courses that participated.

Read more...

Hopkins Student

Actual Hopkins student here with friends who were in that class and witnessed the whole thing from beginning to end. As rosy and clever as this article makes the whole thing sound, it wasn't an all smiles "we gamed the system!" situation.

It was my understanding that this curve for the final had been a long-standing policy of the Professor. There was one student who outright refused to boycott the test (at least the only one who publicly declared his intentions) and was adamant about his taking the final. He had all kinds of moral reasoning about it being the right thing to do and how he had to do it for himself.

Given the nature of the curve, the other students in the class knew that the only way to ace the final and not have to take it was to ensure that not a single student took it. The situation quickly got nasty as the class tried to coerce the student into not taking the exam. The Professor had to step in and expressed his disappointment. Ultimately, the class's repeated statements making him out to be an "asshole" caused him to give in to the pressure.

The Professor's policy changed not because the students figured out a way to game the system, but that this was the first time a student adamantly refused to boycott the final and was nearly physically impeded from doing so.

Read more...

Et

This is correct: the policy was long-standing. There was quite a bit of hostility regarding the ethics of skipping the exam as well. There was a small book's worth of discussion on the topic, some of which was removed from the discussion board due to hostility.

curveBreaker

"...using non-traditional techniques and collaborative learning to surmount the obstacles teachers had put in their way" - (wiki - academic dishonesty)

I respect John Hopkins as an establishment- I hope these student will become leaders one day for their talents, not their politics, their antics.