SAT Strategy by Gender: Men Guess, Women Leave it Blank

(Digital Vision)

To guess or not to guess?  Most students wrestle with this question at least once during their multiple choice test-taking years. A new paper by Harvard economics grad student Katherine Baldiga examines whether men and women approach the issue differently. From the abstract:

In this paper, we present the results of an experiment that explores whether women skip more questions than men. The experimental test consists of practice questions from the World History and U.S. History SAT II subject tests; we vary the size of the penalty imposed for a wrong answer and the salience of the evaluative nature of the task. We find that when no penalty is assessed for a wrong answer, all test-takers answer every question. But, when there is a small penalty for wrong answers and the task is explicitly framed as an SAT, women answer significantly fewer questions than men. We see no differences in knowledge of the material or confidence in these test-takers, and differences in risk preferences fail to explain all of the observed gap. Because the gender gap exists only when the task is framed as an SAT, we argue that differences in competitive attitudes may drive the gender differences we observe. Finally, we show that, conditional on their knowledge of the material, test-takers who skip questions do significantly worse on our experimental test, putting women and more risk averse test-takers at a disadvantage.”

Baldiga’s results might help explain why women often do better in college than their SAT scores would have predicted and raise an important question: Are multiple-choice test scores the best way to fairly “measure aptitude and forecast future achievement”?  Readers, what do you think?  Are SAT tests gender-biased?  Of course, whether or not such gender differences are innate or cultural is a whole other research question.

 

(HT: Market Design)

 

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 38

View All Comments »
  1. alex in chicago says:

    Here is a simple contradictory postulate.

    #1.Being able to read instructions on the test is another skill.
    #2. Instructions contain the information that will tell you the penalty for wrong answers.
    #3. Being able to understand that guessing will improve your performance is a simple calculation that even a fourth grader could do.

    Therefore, not guessing is evidence of inferior knowledge or otherwise an inability to connect ones simple knowledge to a larger scheme.

    Full Disclosure: 35 ACT, Five ’5′s on AP exams (5 taken), 170 LSAT.

    Well-loved. Like or Dislike: Thumb up 13 Thumb down 2
    • rehajm says:

      Sure- I can recall being coached on the ‘educated guess’: if you can eliminate one or more of the answer choices as wrong, it pays to guess, etc.. This is most interesting- are those who are failing to recognize that a guessing strategy can improve your score victims of bias, or is the test recognizing and rewarding those who do- or those who use guessing without the knowledge of it’s advantage, for that matter? Is this a skill we’re trying to identify as aptitude, or is it an unfortunate skew from this type of testing?

      Well-loved. Like or Dislike: Thumb up 5 Thumb down 0
    • Jen says:

      I love that you didn’t take the SATs (according to you), yet put other scores as your “proof”!

      “Therefore, not guessing is evidence of inferior knowledge or otherwise an inability to connect ones simple knowledge to a larger scheme.”

      Someone else above has already pointed out that this isn’t actually true. The guessing penalty on the SAT is 1/4 point (except for the “grid-in” part of a math section which has no guessing penalty, since there aren’t multiple choice answers to choose from). The questions on the test also go from easy to hard in math, the vocabulary sections of the reading and in parts of the writing sections.

      To put it more simply, there’s a lot more to having good strategies on the SAT than your attempted truism.

      Thumb up 2 Thumb down 1
  2. caleb b says:

    While I understand the necessity of standardized tests for college admission, a major problem with the process is that the preparation for the exam is not standardized.

    Some kids, from middle and upper class households, receive tremendous test preparation which can cost hundreds (or even thousands) of dollars. Children from lower class households, generally, cannot afford such a luxury. So Richie Rich goes into the test having prepared for weeks for every type of question that could possibly be asked, while Leon Latchkey takes the test cold. Obviously, Leon isn’t going to score as high as Richie.

    Richie now receives a full-ride merit scholarship based on his score, while Leon does not. Sure, Leon probably qualifies for a Pell Grant, but a Pell Grant only covers basic tuition. Therefore, Leon, who was already at a disadvantage, continues to be at a disadvantage once in college.

    Granted, the issue is more complicated than just that, but current standardized test ignore this aspect.

    Full Disclosure: HS GPA 3.2, ACT 25 No Test Prep, Pell Grant – no other scholarship, College GPA 4.0

    Well-loved. Like or Dislike: Thumb up 8 Thumb down 1
    • Joshua Northey says:

      Yes but those same factors already exist and confound the whole process anyway. If you adjusted test scores for background, then someone with an adjusted test score would need to go through remedial work first because while perhaps naturally more talented, they are at present less prepared. Don’t get me wrong we make horrible use of our human resources in this country, but I feel like standardized tests are our friends not our foes.

      I n college I worked as a tutor at my University’s minority resource learning center and we would have extremely bright migrant kids from say Sub-Saharan Africa, or bright but disadvantaged kids from the inter city US. They were not well served by being “adjusted” into college. What most of them needed was to go back to 4th grade. Now granted since they were bright and older they could probably complete the 4th grade in a couple weeks, but that was where they should have been. Instead we sort of shuffled them through the easiest set of courses we could find and invested a huge amount of tutoring resources in them. Many dropped out anyway. It seemed like a huge waste for a couple of feel good stories to me.

      Full Disclosure HS GPA: 2.3, ACT: 36 (took 2 years early), No Test Prep, College GPA 3.7, Also had only Pell grants in school.

      I came from a pretty crummy background (poor side of town, single parent who was passed out drunk or AWOL most days) and didn’t get into any good schools even with the great test scores. Even somewhere like Reed doesn’t want perfect test scores and a bad attitude. So I ended up going to the U of MN, which was surprisingly good.

      Thumb up 0 Thumb down 0
  3. Ian M says:

    Man are more likely to guess = gender bias in testing? How so?

    The penalty for guessing is often 1/4 of a mark for a 5 choice question. One should ALWAYS guess if they are confident that at least one of the answers is certainly wrong.

    Thumb up 1 Thumb down 0
  4. Jeff F says:

    Actually, Ian, you should always guess if you plan on taking the test more than once. If you take it twice, guessing will create a higher standard deviation to your tests, and most schools only look at the highest score you get (at least that is what schools I applied to stated). Yes, you may get a lower score by guessing, but probabilistically the highest score you get will be higher (if you take it more than once).

    Jeff

    Thumb up 3 Thumb down 0
  5. Joshua Northey says:

    Men are less risk averse than women? You don’t say…

    Coming up at 6 o’clock: Water is Wet a riveting expose’!

    Thumb up 2 Thumb down 0
  6. Joe Dokes says:

    As a male, a teacher, and the father of two girls I have mixed feelings about this research. On the one hand, the fact that it appears that girls do not guess and thus cause their scores to suffer is simply a matter of proper test preparation or simply a change in a test making.

    As a teacher it is clear that it should be trivial to teach people to guess. The key is when you can eliminate at least one clearly wrong answer, it is in your best interest to guess. Having taken hundreds of multiple choice tests over the years, I can say with certainty that I was always able to eliminate at least one obviously incorrect answer. As a result it is always better to guess. Always.

    As a father, it is my responsibility to teach my girls to step up and take risks. It is also my responsibility to ensure that they have the skills necessary to be successful. I also want a fair and level playing field on which they will compete.

    As a male who has watched what I sometimes perceive as a systematic attempt to emasculate males in education I can say that I don’t desire to see any further advantages given to women. Currently women make up 58% of college freshman. It has become clear to me that virtually all changes made to education over the past thirty years have been primarily to the benefit of female students.

    Regards,

    Joe Dokes

    Thumb up 2 Thumb down 1
    • George says:

      “It has become clear to me that virtually all changes made to education over the past thirty years have been primarily to the benefit of female students.”

      That’s the nature of change in institutions — people get disadvantages because the system, as is, disadvantages them somehow. Therefore, change tends to accrue to the benefit of the disadvantaged. Whether that’s deserved is a different question, but if you don’t believe that women were disadvantaged (in education or generally) thirty years ago, then you weren’t paying attention.

      Thumb up 3 Thumb down 0
  7. Brian Gulino says:

    What an interesting paper Ms. Baldiga wrote. How small a penalty for a wrong guess? If a correct answer is worth 4 points and you are choosing among 3 answers, a 1 point penalty would reward guessing and a 2 point penalty would penalize guessing. Very odd that this is not made explicit in the abstract and that none of the commenters mentioned it.

    Since the penalty determines the strategy, I read the paper where I found this incorrect analysis of test taking strategies:

    For instance, on the SAT, a long-time staple of college admissions in many countries, answering a multiple-choice question always yields a weakly positive expected value. There are five possible answers; one point is given for a correct answer, 1/4 of a point is lost for an incorrect answer, and no points are awarded for a skipped question.

    Ms. Baldiga incorrectly concludes:

    Even when he is unable to eliminate any of the possible answers, a risk neutral test-taker maximizes his expected score by answering the question.

    Well, no. Random guesses on a hundred question test with 5 choices per question yields (on average) 20 correct answers for 20 points minus 80 errors at 1/4 point penalty for -20 points for a net of zero points. Guessing on SAT’s is risk neutral.

    Ms. Baldiga cites various studies of gender specific behavior on tests. Its remarkable how disinterested the test givers are in optimal guessing strategies, a disinterest seemingly shared by Ms. Baldiga even though she’s writing a paper about it. As a test taker, only recently a test giver, I have always been interested in optimal test taking strategies. For the SAT’s my strategy was:

    Go through the test answering all questions I am sure of. On the questions I wasn’t sure of, lightly mark all the answers I knew were wrong on the answer sheet. Use the remaining time to revisit those questions. In the last minute of the test, guess among all the answers that were not lightly marked.

    Thumb up 3 Thumb down 1
    • Jeffrey L. says:

      Yes, they do make that incorrect conclusion in the introduction, but their version of the SAT that they gave the students only had 4 questions, but there was still only a 1/4 point penalty. So even though the 5-point SAT is risk neutral, this test was rewarding guessing.

      Meanwhile, the words they used to frame it as an SAT said that it would be “scored like an SAT.” I took the ACT, and the only things I know about the difference is that SAT uses bigger numbers to say the score, and that you are penalized for guessing. Were women less likely to go, “Oh, they mean ‘similar’ but there is a subtle difference that puts guessing in my favor!”

      But even though men guessed more, they scored the same as the women. Are men worse at guessing? Or was the test not long enough to make guessing matter? I liked the pilot session where there were the 4 questions, then a 5th that said, “I’m not sure, but I would guess ____.” Women used that answer a lot more than the men. Without considering the guesses, men did a bit better, but when the guesses were factored in they scored the same.

      Probably need a bigger test, but I want to say that these women are either better guessers, or are just less sure of themselves even when they are right. I think there’s a moral somewhere here to take away from this.

      Thumb up 0 Thumb down 0
  8. Shane says:

    If they were to increase the cost of a wrong answer, might we not see female students converging with, or overtaking, the male students again? Perhaps the negative marking at the moment is not negative enough. The cost is enough to deter risk-taking among the women, but not enough to punish risk-taking by the men?

    Thumb up 0 Thumb down 1