How to Game a Grading Curve

Students in three of Professor Peter Fröhlich‘s computer programming classes at Johns Hopkins University recently devised a method to game their final grades.  Frolich grades exams on a curve — the highest grade in the class, whatever it may be, becomes 100 percent, and “everybody else gets a percentage relative to it.”  So students collectively planned a boycott:

Because they all did, a zero was the highest score in each of the three classes, which, by the rules of Fröhlich’s curve, meant every student received an A.

“The students refused to come into the room and take the exam, so we sat there for a while: me on the inside, they on the outside,” Fröhlich said. “After about 20-30 minutes I would give up…. Then we all left.” The students waited outside the rooms to make sure that others honored the boycott, and were poised to go in if someone had. No one did, though.

Catherine Rampell discusses the strategy:

This is an amazing game theory outcome, and not one that economists would likely predict…

In this one-off final exam, there are at least two Bayesian Nash equilibria (a stable outcome, where no student has an incentive to change his strategy after considering the other students’ strategies). Equilibrium #1 is that no one takes the test, and equilibrium #2 is that everyone takes the test. Both equilibria depend on what all the students believe their peers will do.

If all students believe that everyone will boycott with 100 percent certainty, then everyone should boycott (#1). But if anyone suspects that even one person will break the boycott, then at least someone will break the boycott, and everyone else will update their choices and decide to take the exam (#2).

The problem is that Nash equilibrium theory alone doesn’t tell us what the students are more likely to do. Economists would say that the first equilibrium, where no one takes the exam, is unlikely to result because it is not “trembling hand perfect,” an idea that helped win Reinhard Selten win the Nobel Memorial Prize in Economics.

Fröhlich was impressed by the students’ scheme. “The students learned that by coming together, they can achieve something that individually they could never have done,” he wrote in an e-mail. “At a school that is known (perhaps unjustly) for competitiveness I didn’t expect that reaching such an agreement was possible.”  He has, however, revised his grading policy to prevent future gaming.

(HT: Sarah Martin)

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 46

View All Comments »
  1. tung bo says:

    The students are also betting that their cohesiveness and game theoretical thinking will impress Prof. Frohlich. They stood to lose if the Professor took a legalistic approach: since none of the student participated in the exam by going into the room, the Professor can treat that as a forfeit. That means no grades for any student or failling grades for forfeits.

    This risk should force some and then most student to defect given a large enough class. With a small class, it was possible to enforce nonparticipation.
    Yet, the Professor can also offer a make-up take home test to replace the in class test. Without the public mutual monitoring, almost certainly some will defect.

    Ultimately, the students were betting that Prof. Frolich was a ‘nice’ person and would not follow these other alternatives.

    Well-loved. Like or Dislike: Thumb up 54 Thumb down 9
    • Matt says:

      In high school I had a math teacher who had a similar grading curve policy for the final. A bunch of hands went up after she finished explaining it, at which point she amended it by saying that if everyone did very poorly (e.g., scored 0s) she would nullify the test and make us take a new one (and half the hands went down).

      It all comes down to how much one thinks the teacher will appreciate the effort.

      Thumb up 4 Thumb down 1
    • Neil (SM) says:

      Perhaps the prof may be bound by his own syllabus — at least the first time before it is revised for such an event!

      Well-loved. Like or Dislike: Thumb up 5 Thumb down 0
    • mike says:

      Tung Bo- I am not sure that you are correct. The course syllabus spells out the rules of engagement and the course grading “contract.” To re-write rules for punitive gain because the Prof did not like the outcome would be a challenge. Moreover, I did like how the students managed to collective escape the prisoner’s dilemma (or Julien Edney’s Nuts Game) problem.

      Well-loved. Like or Dislike: Thumb up 10 Thumb down 3
  2. Jaime says:

    It seems to me that this result was in all likelihood triggered by some previous action (on behalf of the teachers, university, etc) that removed any incentive for competition in the class as this is one of two mechanisms I see that would lead to this equilibrium. Alternatively it is possible some group of students with little chance of getting a high grade coerced their peers to take part in the plan.

    This latter mechanism is less likely and not stable, as students actually have an incentive (that increases as more students are coerced into getting a 0%) to stand up to those that want to attempt the 0% grade strategy as the resisting student’s grade is bound to see a greater difference compared to those other students in their class and hence end up with a more outstanding degree than his/her peers.

    The former mechanism could be, for example, that the given class would not count for the final grade of the course/degree but a pass was still required to successfully complete the course/grade. This situation creates a stable equilibrium where all students are highly incentived to carry out such a strategy as there is no comparable gain between the “0%, 0 effort strategy” and “each student individually studying to get a passing grade” strategy, while there is a clear cost difference (in effort) that pushes the group towards the consensus arrived at in the anecdote.

    Would be nice to get some background on this.

    Thumb up 3 Thumb down 0
    • J says:

      Basically none of these equilibrium theories apply because we all knew what the other students were going to do beforehand, and on the day of the test.

      I was in one of his classes, and I got 100 on my final using this strategy. First of all, I’m pretty sure Peter mentioned this possibility at the beginning of the year after he told us how his scale worked. If he hadn’t I don’t think anyone would have believed it would actually worked.

      The reason we could organize this is that the class was using an online discussion website thing where everyone in the class was registered, and you could ask questions about homework or whatever and get an answer from your fellow students or the professor. So it was pretty easy for someone to post the idea and see if anyone was opposed to it. Most people in the class were stressed about finals and were happy to have a guaranteed 100 regardless of how they were doing in the class at the time, so they agreed to the scheme. A few other people didn’t really want to do it but were willing to go along with it to help out their peers and avoid forcing a bunch of people to take the test. Then there was one kid who refused to skip the test because of moral and philosophical reasons. Somehow someone eventually convinced him to go in, take the test, but not hand it in, while everyone else stood outside and watched. It was pretty weird and I felt pretty bad for putting him in such an uncomfortable position, but he didn’t turn it in so we all got the 100.

      Thumb up 4 Thumb down 1
  3. Joe says:

    Should they have all gotten 100%? I think that they should have gotten an undefined 0/0 if that were the maximum grade.

    Well-loved. Like or Dislike: Thumb up 14 Thumb down 4
    • salviati says:

      That is a loophole wide enough to drive a truckload of F’s through. I love it. All the professor need do is to change the syllabus to read that he will “determine their percentage grade based on the ratio out of the highest score” and is thus free to determine that 0/0 = 0%

      Thumb up 0 Thumb down 2
      • Asaf says:

        0/0 is undefined… Not something a computer scientist would do :)

        Thumb up 0 Thumb down 0
      • Asaf says:

        What I meant to say is that “undefined” does not equal to “you can define it however you like”, allowing division by zero leads to a contradiction and is inconsistency.

        Thumb up 0 Thumb down 0
  4. Ana Bee says:

    One other thing to take under consideration the “dude, don’t be an asshole” cause, which is sufficient to persuade a person not to break contract, when uttered near the classroom door.

    Well-loved. Like or Dislike: Thumb up 18 Thumb down 1
    • Travis Idol says:

      Exactly. Game theory in its simplest form assumes the participants don’t know or can’t directly influence the actions of the other players. In fact, the classic prisoner’s dilemma works precisely because the defendants are kept separate. A boycott (think union strike) works because the participants stand around the entrance and attempt to discourage anyone thinking of crossing the picket line.

      Thumb up 1 Thumb down 0
  5. Lou says:

    A more sophisticated curve would raise the median or mean to a desired value (say 75 or 80%). Thus everyone would get the same lower grade and remove the incentive for the top students to participate.

    Well-loved. Like or Dislike: Thumb up 7 Thumb down 0
    • Peter says:

      Lou, I was going to do the same thing. If the curve were based on the median score there should be, by definition, 50% of the class than can expect to receive a better grade. This would restore the separating equilibrium, by ensuring one student has an incentive to defect.

      Thumb up 1 Thumb down 0
  6. Khurram Makhdumi says:

    If nobody entered the exam hall then they don’t get zeroes, they were absent. To get zero, one must enter the exam hall and at least write name on the answer sheet. Given every student has an individual answer sheet, observing the divergent attitude directly is harder, thus the incentive to diverge and the more persistent equilibrium, i.e. everybody takes an exam.

    Well-loved. Like or Dislike: Thumb up 9 Thumb down 4
    • jono gabono says:

      Not really, on my University there is no such grading as “absent”, if you where absent you get a big fat zero.

      Thumb up 1 Thumb down 0
  7. Enter your name... says:

    The students took a significant risk, because the instructor had another option: to refuse to give them grades and instead assign them “incompletes”.

    Well-loved. Like or Dislike: Thumb up 11 Thumb down 3
  8. Tom Fox says:

    It seems to me like they all should have received an incomplete rather than a zero. Did he take attendance?

    If this was my class trying to “game” my system I would have gamed them right back and said that there is a difference between not taking and not passing the exam. That would force them to all fill in “A” or something like that. Then we would see how much they trust each other.

    Well-loved. Like or Dislike: Thumb up 13 Thumb down 6
    • J says:

      The only reason this worked is the professor basically told us this was an option when he told us how his grading scheme worked. He doesn’t think tests are a good way of evaluating performance so he was ok giving us all 100s, especially since the final wasn’t worth a huge percentage of our final grade. And I’m pretty sure the only reason he changed his policy was because the rest of the CS department was furious at him.

      Well-loved. Like or Dislike: Thumb up 8 Thumb down 2