The Art of SATergy

My son took the SSAT exam this past Saturday. And while I was sitting in the Choate athletic facility waiting for him to finish, I remembered that Avinash Dixit and Barry Nalebuff‘s new book, The Art of Strategy, has a great example concerning standardized testing. Game theory is so powerful it can help you figure out the correct answer without even knowing what the question is.

Consider the following question for the GMAT (the test given to MBA applicants). Unfortunately, issues of copyright clearance have prevented us from reproducing the question, but that shouldn’t stop us.

Which of the following is the correct answer?

a) 4p sq. inches

b) 8p sq. inches

c) 16 sq. inches

d) 16p sq. inches

e) 32p sq. inches

O.K., we recognize that you’re at a bit of a disadvantage not having the question. Still, we think that by putting on your game-theory hat you can still figure it out.

Before reading their analysis, take a shot at trying to reason your way to the correct answer.

Here’s what they said:

The odd answer in the series is c. Since it is so different from the other answers, it is probably not right. The fact that the units are in square inches suggests an answer that has a perfect square in it, such as 4p or 16p.

This is a fine start and demonstrates good test-taking skills, but we haven’t really started to use game theory. Think of the game being played by the person writing the question. What is that person’s objective?

He or she wants people who understand the problem to get the answer right and those who don’t to get it wrong. Thus wrong answers have to be chosen carefully so as to be appealing to folks who don’t quite know the answer. For example, in response to the question: “How many feet are in a mile?” an answer of “Giraffe,” or even 16p, is unlikely to attract any takers.

Turning this around, imagine that 16 square inches really is the right answer. What kind of question might have 16 square inches as the answer but would lead someone to think 32p is right? Not many. People don’t often go around adding p to answers for the fun of it. “Did you see my new car — it gets 10p miles to the gallon.” We think not. Hence we can truly rule out 16 as being the correct solution.

Let’s now turn to the two perfect squares, 4p and 16p. Assume for a moment that 16p square inches is the correct solution. The problem might have been: “What is the area of a circle with a radius of 4?” The correct formula for the area of a circle is pr2. However, the person who didn’t quite remember the formula might have mixed it up with the formula for the circumference of a circle, 2pr. (Yes, we know that the circumference is in inches, not square inches, but the person making this mistake would be unlikely to recognize this issue.)

Note that if r = 4, then 2pr is 8p, and that would lead the person to the wrong answer of b. The person could also mix and match and use the formula 2pr2, and hence believe that 32p or e was the right answer. The person could leave off the p and come up with 16 or c, or the person could forget to square the radius and simply use pr as the area, leading to 4p or a. In summary, if 16p is the correct answer, then we can tell a plausible story about how each of the other answers might be chosen. They are all good wrong answers for the test maker.

What if 4p is the correct solution (so that r = 2)? Think now about the most common mistake: mixing up circumference with area. If the student used the wrong formula, 2pr, he or she would still get 4p, albeit with incorrect units. There is nothing worse, from a test maker’s perspective, than allowing the person to get the right answer for the wrong reason. Hence 4p would be a terrible right answer, as it would allow too many people who didn’t know what they were doing to get full credit.

At this point, we are done. We are confident that the right answer is 16p. And we are right. By thinking about the objective of the person writing the test, we can suss out the right answer, often without even seeing the question.

Now, we don’t recommend that you go about taking the GMAT and other tests without bothering to even look at the questions. We appreciate that if you are smart enough to go through this logic, you most likely know the formula for the area of a circle. But you never know. There will be cases where you don’t know the meaning of one of the answers or the material for the question wasn’t covered in your course. In those cases, thinking about the testing game may lead you to the right answer.

If you want a fun way to learn a ton of useful game theory, this is the book for you. How good is it? Steve Levitt has a blurb on the book saying it’s so good, he read it twice.


Rachael

Yep, that's the answer I got.

A simpler version of the reasoning, which isn't as game-theoretically watertight as that given here but which used to serve me well at school, is to pick the answer which has most in common with the other answers, so all the other answers deviate from it in minimal ways. So, in this example, all the answers have a 16 or a pi in them, so the correct answer is the one with both a 16 and a pi.

Similarly with spelling questions: the choice of answers will often be something like "necessary, necesary, neccessary" and even if you don't know the correct answer, you'd pick the first, because it's equal in edit distance from the other two. Whereas if the last one were right, the other two would be suboptimal decoy answers, since one would be some distance from the correct answer and the other would be even further from it in the same direction.

Adam Bee

It didn't seem fair to me that they allowed themselves to use their correct knowledge of the formula when figuring out the answer. They also shouldn't have spent so much time reverse engineering what the question is.

In any realistic situation, the test-taker will know what the question is and not know what the answer is.

The only part of this that I found useful was the ruling out of c. because it is so obviously much different. The rest is just BS filler.

Kevin

Unfortunately this would have no use on the exam since you used the formulas that would probably be needed to answer the original question to root out the potentially wrong ones...Besides the fact that using such logic would take far too long.

Anna

You didn't even need to go through all that analysis. It's just a matter of picking the most common of the different elements. Most of the choices have Pi. Rule out C. The only number that is a choice more than once is 16. 16pi.

You don't even have to know the first thing about circle formulas, and I would bet that most people reading this and guessing the right answer went about it my way.

chappy

Well I arrived at the correct answer, but I think my logic was not as involved. I agreed with the logic that 16 sq. inches seemed to odd, but then I thought why would they leave it in...because 16(pi) square inches is the right answer and they are trying to 'catch' people that aren't looking at the units.

jz

so, I'm not as smart as I'd prefer to think.

adriano

Is either 4pi or 16pi depending on the radius, 2 or 4 respectively

adriano, Italy

Matt C

It's even easier than that. One thing the authors missed: the only answer without a pi is 16. Since each wrong answer usually represents the inclusion of a single common mistake, and we're pretty sure the answer has pi after it, 16pi makes the most sense. If the answer were 16pi and the only number without a pi were 8, that number would only appeal to people who made TWO mistakes (using the wrong formula AND dropping the pi). Unless they're saying the pi-less answer was a throwaway, ignoring the fact that the choices include both 16 and 16pi is ignoring a very relevant and helpful piece of information.

F

There is an SAT guide written specifically for smart kids on how to get a better score called "Up Your Score." It was written by a bunch of kids from Ithaca HS in Ithaca NY. They have a guessing strategy that is much simpler but based in the same general theory, that still gets you to 16pi. Their strategy is to pick the answer that is most like the other answers.

So looking at the answers above, you would immediately eliminate c) because all the other answers contain pi. However, 16 also appears in 2 answers (none of the other numbers appear more than once), so you would guess 16pi as the right answer. That takes about 2 seconds to do- much quicker than Dixit & Nalebuff's strategy, which although more thorough, wastes precious time that you could be using to figure out questions that you don't need to blindly guess an answer.

Dan

You can immediately throw out c because it doesn't have a pi, but it also indicates that 16 is in the answer. You see this all the time in standardized tests. Also if you add the coefficient of the first answer to itself you get the second answer and if you square it you get the 3rd. You see this a lot as well and it usually means that one of the two are correct. Since we are talking about squares and with the help of answer C, you can reasonably assume the answer is D. It's not as detailed as your description, but a hell of a lot faster. I tihnk it would be fun to take the SATs without the questions.

sean s.

I have to agree with some previous commentators. The technique would not be usable in an actual test because 1) to make it work you need to understand the common errors test-makers are looking for. If your understanding is that deep, you don't need the game-theory. And 2) it would take way to long to answer a significant number of questions this way.

It is interesting to think of tests this way, and to try to figure out the correct answer without seeing the question. However, could there be alternative questions which lead to different correct answers within the same set of choices? Is 'd' the exclusively correct answer?

Mike

Ahhhh, Up Your Score. I usually say The Princess Bride is my favorite book ever, but Up Your Score might top it. My father bought it for me when I was in middle school because for some reason I got to take the SAT's in 8th grade to see if I qualified for some smart-kids program at Johns Hopkins. I didn't. But I learned a ton from that book, and read it probably once a year throughout high school. I'd bet it added 50 points to my SAT score. It's such a smart, well-written book, with some pretty good humor in it too, to keep it interesting. And it's probably going to improve one's SAT score by more than a Kaplan course (or the like). When you figure it's also roughly 1/50th of the cost, it's a no-brainer!

Grant

I like the idea that this broken, farcical test can be broken, but I believe any high schooler with an understanding of game theory would probably do OK on his/her own.

Mike

And #2 and #3 are correct... if you used the correct formula to inform this decision making process, you probably should have just used it to solve the problem in the first place. Hence, the Up Your Score strategy (which I call the Sesame Street "one of these things is most like the other") is the quickest and easiest.

Up Your Score: 1, Freakonomics: 0

charles

If you're using good test taking strategy and skipping what you don't know this method is indeed useful, and there is plenty of time available contrary to what the posters have posted. It takes much longer to read it than think it through.

Mike M

It's amazing how many simple, little standardized test tricks there are. When used together, you do not have need anywhere near a mastery of the material to ace the exam.

Of course it helps if you have above average intelligence to make connections that the average person will miss.

I remember teaching myself math in high school by reverse solving the questions on the exams (I wasn't a good student and often slept through class).

Christopher

You really need to see other questions and know enough correct answers for them to get a picture of how the test writer's mind works.

I remember many a test that I didn't study for, and I remember doing these same mind tricks mentioned here.

I happened to answer 16 pi squared too simply because there was another 16 and the rest had pi (a technique many have mentioned in this forum).

However, if I were a test writer I may deliberately put those sorts of answers in there to fool someone. If the correct answer here were 4 pi squared, I may very well put in the same C and D answers to fool the guessers.

Mark

Those of you who say this isn't useful in a real test could not be more wrong! The point isn't to get directly to the answer without looking at the question, it's to use available information to eliminate incorrect answers, to confirm your answer, or to select an answer even when you're not 100% sure that it's correct. It's extremely useful, and any test prep course will cover this sort of thing to a greater or lesser extent. In my own experience, learning tricks that put you in the head of the question writer, or that make use of available information not directly communicated to the test taker (for instance, knowing which Greek and Latin roots have good and bad connotations) was the difference between a pretty good score and a great score.

Standardized tests are very teachable. I imagine this is part of the reason that ETS stopped trying to pretend the SAT is an "aptitude" test.

mccxxiii

I think a lot of the people who have commented here are missing the point of this column.

If you were taking an actual test, you would be able to read the questions, and you wouldn't have to take such a long time to go through all the "game theory" steps every time. The point of making it so long and drawn-out here is only to demonstrate the concept.

If you're at all prepared for the test, 1/3 of the questions will be completely easy, 1/3 will be a bit of a challenge, and 1/3 will require serious attention. Even of those in the last category, only a few would require this level of examination, and it would still be easier than this because you would know the question.

kip

Has anyone ever tested this methodically? All the support for these methods seem to be anecdotal, or cherry-picked questions. As a general rule, does it help you?

Here's what I think you could do to test this:
Give a bunch of high school seniors an old SAT test, with all the questions removed. See how well they do in the same time limit as the general test, or with no time limit. Let some of them be taught the basic principles of smart guessing, give others no preparation. For reading comprehension questions, you could even have a group that actually gets the full questions, but does not get to read the passage about which the questions are asked. Have a control group that has no questions or answers, just a bubble sheet (this would be the random guessing group).

My hypotheses:
* Intelligent guessing will yield a slight (but statistcally significant) improvement over the control group.
* Regressions against the students' actual SAT scores (like, when they took the test for real) will reveal positive correlation between intelligent guessing performance and full knowledge performance. (In other words, if you are already good at math, you will be better at guessing the answer to math questions even without having the questions available).
* Intelligent guessing will take a long time (so the students who do not have a time limit will do far better than those who do).
* Intelligent guessing will be more effective in math problems than in vocabulary/reading questions.

Another hypothesis that could be tested in a different experiment: Intelligent guessing would be much more effective against non-standardized tests (i.e. tests that high school teachers create themselves) than against standardized tests such as SAT, ACT, GRE, etc.

Read more...