SAT Strategy by Gender: Men Guess, Women Leave it Blank

(Digital Vision)

To guess or not to guess?  Most students wrestle with this question at least once during their multiple choice test-taking years. A new paper by Harvard economics grad student Katherine Baldiga examines whether men and women approach the issue differently. From the abstract:

In this paper, we present the results of an experiment that explores whether women skip more questions than men. The experimental test consists of practice questions from the World History and U.S. History SAT II subject tests; we vary the size of the penalty imposed for a wrong answer and the salience of the evaluative nature of the task. We find that when no penalty is assessed for a wrong answer, all test-takers answer every question. But, when there is a small penalty for wrong answers and the task is explicitly framed as an SAT, women answer significantly fewer questions than men. We see no differences in knowledge of the material or confidence in these test-takers, and differences in risk preferences fail to explain all of the observed gap. Because the gender gap exists only when the task is framed as an SAT, we argue that differences in competitive attitudes may drive the gender differences we observe. Finally, we show that, conditional on their knowledge of the material, test-takers who skip questions do significantly worse on our experimental test, putting women and more risk averse test-takers at a disadvantage.”

Baldiga’s results might help explain why women often do better in college than their SAT scores would have predicted and raise an important question: Are multiple-choice test scores the best way to fairly “measure aptitude and forecast future achievement”?  Readers, what do you think?  Are SAT tests gender-biased?  Of course, whether or not such gender differences are innate or cultural is a whole other research question.

 

(HT: Market Design)

 

Joe Dokes

As a male, a teacher, and the father of two girls I have mixed feelings about this research. On the one hand, the fact that it appears that girls do not guess and thus cause their scores to suffer is simply a matter of proper test preparation or simply a change in a test making.

As a teacher it is clear that it should be trivial to teach people to guess. The key is when you can eliminate at least one clearly wrong answer, it is in your best interest to guess. Having taken hundreds of multiple choice tests over the years, I can say with certainty that I was always able to eliminate at least one obviously incorrect answer. As a result it is always better to guess. Always.

As a father, it is my responsibility to teach my girls to step up and take risks. It is also my responsibility to ensure that they have the skills necessary to be successful. I also want a fair and level playing field on which they will compete.

As a male who has watched what I sometimes perceive as a systematic attempt to emasculate males in education I can say that I don't desire to see any further advantages given to women. Currently women make up 58% of college freshman. It has become clear to me that virtually all changes made to education over the past thirty years have been primarily to the benefit of female students.

Regards,

Joe Dokes

Read more...

George

"It has become clear to me that virtually all changes made to education over the past thirty years have been primarily to the benefit of female students."

That's the nature of change in institutions -- people get disadvantages because the system, as is, disadvantages them somehow. Therefore, change tends to accrue to the benefit of the disadvantaged. Whether that's deserved is a different question, but if you don't believe that women were disadvantaged (in education or generally) thirty years ago, then you weren't paying attention.

Brian Gulino

What an interesting paper Ms. Baldiga wrote. How small a penalty for a wrong guess? If a correct answer is worth 4 points and you are choosing among 3 answers, a 1 point penalty would reward guessing and a 2 point penalty would penalize guessing. Very odd that this is not made explicit in the abstract and that none of the commenters mentioned it.

Since the penalty determines the strategy, I read the paper where I found this incorrect analysis of test taking strategies:

For instance, on the SAT, a long-time staple of college admissions in many countries, answering a multiple-choice question always yields a weakly positive expected value. There are five possible answers; one point is given for a correct answer, 1/4 of a point is lost for an incorrect answer, and no points are awarded for a skipped question.

Ms. Baldiga incorrectly concludes:

Even when he is unable to eliminate any of the possible answers, a risk neutral test-taker maximizes his expected score by answering the question.

Well, no. Random guesses on a hundred question test with 5 choices per question yields (on average) 20 correct answers for 20 points minus 80 errors at 1/4 point penalty for -20 points for a net of zero points. Guessing on SAT's is risk neutral.

Ms. Baldiga cites various studies of gender specific behavior on tests. Its remarkable how disinterested the test givers are in optimal guessing strategies, a disinterest seemingly shared by Ms. Baldiga even though she's writing a paper about it. As a test taker, only recently a test giver, I have always been interested in optimal test taking strategies. For the SAT's my strategy was:

Go through the test answering all questions I am sure of. On the questions I wasn't sure of, lightly mark all the answers I knew were wrong on the answer sheet. Use the remaining time to revisit those questions. In the last minute of the test, guess among all the answers that were not lightly marked.

Read more...

Jeffrey L.

Yes, they do make that incorrect conclusion in the introduction, but their version of the SAT that they gave the students only had 4 questions, but there was still only a 1/4 point penalty. So even though the 5-point SAT is risk neutral, this test was rewarding guessing.

Meanwhile, the words they used to frame it as an SAT said that it would be "scored like an SAT." I took the ACT, and the only things I know about the difference is that SAT uses bigger numbers to say the score, and that you are penalized for guessing. Were women less likely to go, "Oh, they mean 'similar' but there is a subtle difference that puts guessing in my favor!"

But even though men guessed more, they scored the same as the women. Are men worse at guessing? Or was the test not long enough to make guessing matter? I liked the pilot session where there were the 4 questions, then a 5th that said, "I'm not sure, but I would guess ____." Women used that answer a lot more than the men. Without considering the guesses, men did a bit better, but when the guesses were factored in they scored the same.

Probably need a bigger test, but I want to say that these women are either better guessers, or are just less sure of themselves even when they are right. I think there's a moral somewhere here to take away from this.

Read more...

Shane

If they were to increase the cost of a wrong answer, might we not see female students converging with, or overtaking, the male students again? Perhaps the negative marking at the moment is not negative enough. The cost is enough to deter risk-taking among the women, but not enough to punish risk-taking by the men?

Owen

People have already done research showing High School GPA is the best to “measure aptitude and forecast future achievement”

Rik

I thought the different SAT taking strategies of boys and girls and their willingness to guess was already known. I can remember this "fact" being thrown around in my SAT prep classes in the 90s.

Ollie

I attended a high school in a foreign country. A significant number of our students have taken the SAT tests and consequently attended US universities throughout the years. Though the school doesn't keep statistical data, from what teachers and counselors have shared, overall women here perform better. I was surprised when I read statistical data which stated that women in the US performed better in the verbal section and men in the math section, because in our case it was usually women performing slightly better on both sections. Anyway, we were all taught to never leave a blank answer, despite the penalty points. Of course, I don't have enough data to claim that, but do you think the differences may have something to do with the type of educational system at hand, teachers' training and advice etc.? Maybe even parents' expectations play a role.

I think multiple-choice tests are not the best indicator of later success but they are fairly accurate and probably the best we have at hand now.

Read more...

robin marlowe

Woops, upon second thought almost all. As to the issue at hand, I once took a standardized test in "social studies." The thing is, I never took a college course in the subject. Bought a book, made educated guesses. That's what someone suggested that I do. Low and behold- I did well- without any formal training. So women can be taught to behave otherwise than they would. Does this not mean that men can learn as well. I do have reason to think so.

Adam

If you know at least 1 answer is wrong, you should guess between the others. If girls aren't doing this, they should. I don't think the test needs to be changed. If there are 20 questions with 5 possible choices and you guess randomly, you get 4 right (4pts), 16 wrong (-.25 * 16 = -4) and you end up with 0 (current scoring). If you can eliminate 1 choice from each, you should get 5 right (5 points) and 15 wrong (-.25 * 15 = 3.75) and you end up with 1.25. This improvement expresses your partial knowledge on the subject.

If girls did better on the verbal section of the test would we call the test gender-biased or would we just say that the boys need to do better there?

Dorado

This doesn't surprise me at all. Women are the more risk-averse of the sexes. The main difference is testosterone influences aggression and risk-taking. Since obviously men produce much more then women, a man is more likely to think to himself: "I don't know the answer but I have a 25% chance of being right" when confronted with a question he is unsure of. A female test-taker would worry more about the repercussions (losing points, failing the exam, etc) than the possibility of being right. Basically, testosterone makes men more likely to go against the grain and break the mold so to speak which explains why most societies, cultural institutions, inventions, have all been created/founded by men.