The Dangers of N=1

INSERT DESCRIPTIONPhoto: Opo Terser

From a reader named Mike Friedman:

I realized this morning that my daily behavior has been modified by a data point of one. I thought it would be interesting to you. Oh, and apologies in advance for the pun:

For the most part, my morning routine is the same every day. I shower, towel off, comb my hair, yada yada yada. A few weeks ago, while toweling off, I noticed a rather large spider on my towel. Being a little squeamish around spiders, I quickly threw down the towel and disposed of the intruding arachnid. I forgot about the “insect-dent” and went about my day.

Cut to this morning. After getting out of the shower and grabbing my towel, I experienced a moment of self-awareness and realized I was subconsciously checking my towel for spiders before applying it to my face. I realized in that moment of self-awareness that my behavior had been influenced by a single data point.

Consider the numbers: At 35 years old, I figure I have conservatively showered well over 12,500 times in my life, and I have only found one spider in my towel. And yet, for the last three weeks or so, I have been checking my towel every time I shower.

It got me thinking: How often do we allow our behavior to be influenced by single data points. Are there any positive examples?

Mike’s e-mail appealed to me because it touches on a lot of things that have been discussed here over the years, including recency bias, black swans, and the strange case of Baby Emily, whom we wrote about here:

In the early 1980’s, a group of psychologists and linguists banded together to write Narratives From the Crib, a study of how children acquire linguistic skills. Narratives was built around the speech patterns of one child, a 2-year-old girl. Her parents had noticed that she often talked to herself in the crib after they said good night and left her room. They were curious to know what she was saying, so they began to record her chatter. They turned on the tape recorder while they were tucking her in and then left it running.

Eventually they gave the tapes to a psychologist friend, who shared it with her colleagues. The big surprise to these experts was that the girl’s speech was far more sophisticated when she was alone than when she was speaking with her parents. This finding, as Malcolm Gladwell would later write in The Tipping Point, “was critical in changing the views of many child experts.”

The 2-year-old girl in question was referred to as Baby Emily. Her full name? Emily Oster. In retrospect, it would appear that Narratives From the Crib suffers what researchers call an “n of 1” problem, with “n” representing the size of the sample set — a problem that is gravely exacerbated when the one subject turns out to be … well, a good bit brighter than average. Studying how children learn to talk by observing Baby Emily may be a bit like studying how children learn to play golf by studying Tiger Woods.


Alice

I have a big spider phobia - so much so that I had to look away from the picture on the screen as I scrolled down in order to read this article.

When I was about 8, I got out the bath and put on my white bathrobe. As I pulled it around my body I noticed a black splodge on the inside of the robe, it was a spider. I threw off the robe and screamed the house down. Even now, at 33, I'll look down both sides of a robe before wrapping it around me.

Add to that - the whole spider in the bedroom incident. There it was on the ceiling just as I walked into the room and it scared me witless. For ever more I'll check the ceiling of most rooms but mostly bedrooms for spiders, even my current bedroom, which has been spider free (AFAIK) the entire time I've been there.

Reading this article and comments though, I was tempted to go and check all my clothes I'm currently wearing and all my towels back home JUST in case.

The irrational mind...

Read more...

Ruth

When I was in junior high or high school, I read a story about how a boy had been raised as a girl after a botched circumcision. I was horrified. When my son was born, there was discussion in the medical community about how circumcision was an unneccessary procedure. I did not have my son circumcised, since my religion didn't require it either. My obstetrician was enthusiastic about my decision, but sometimes I wonder if it was right. We were transferred several times, and there were a couple of communities where his differences led to teasing and ostracism. Meanwhile, the boy raised as a girl rebelled and went back to being a boy, and has since committed suicide. I can only imagine his family's pain. Anecdotal evidence can be very convincing to humans, even when the evidence isn't statistically convincing.

Homer

should'nt it be n=1 rather than N=1 ?

Clyde Kahrl

Wow. I just read a WHOLE PILE of stuff about single data points and so forth and NOT ONE pointed out the FUNDAMENTAL ISSUE.

What IS the fundamental issue?

The fundamental issue is: HYPOTHESIS TESTING.

Hypothesis testing ONLY WORKS IF YOU CAN DESCRIBE THE UNIVERSE WITH THE COMPLETE CERTAINTY OF SEVERAL HYPOTHESES !!!!!!!!!

So if the spider in the towel doesn't figure into your prior hypotheses, THEN ALL OF YOUR PRIOR DATA POINTS ARE IRRELEVANT.

Statistics cannot prove anything: Statistics can only DISPROVE THINGS.

Rwaz

Every morning I step out for the paper. Routine. Half asleep. A few years ago I stepped out on a sheet of ice and my feet went above my head and my head bounced off the concrete. A few inches and my head would have hit the ledge. I've felt ever since that I'm on borrowed time.

Now, even in the middle of July, I stop and look before I take the first step.Hopefully my n of 1 can help others and they'll learn with an n of 0.

How many are resting peacefully because of a few inches and because the n of 1 never happened?

Anna

It doesn't matter if it's logical or not. You're going to do it anyhow, because that's how your brain is programmed. You never decided to check your towel, you just did it automatically. Our reptilian brain works before our analytical brain kicks in. We think we are mostly logical, but in reality we're mostly superstitious, especially when it comes to very emotional experiences, like the fright of seeing a spider.

Carl

It seems that some of this is cost/benefit, too.

If the N=1 example demonstrates that by simply shaking a towel, we can avoid a risk, then we apply that lesson. I suspect that if the cost or risk avoidance were higher, (keeping all towels in shrinkwrap?) the N=1 example wouldn't cause a behavior change.

John S.

Learning — even over-learning — from n = 1 can be better than not learning from n = 1 or 2 or 3 or 4 or . . . .

Witness the non-learning of bankers Muhammad Yunus experienced when trying to get them to make loans to poor people.

http://www.bankertothepoor.com/bankertothepoor/

(Bankers come into the story at the end of the third page.)

Allen

Number 19 is on the right track.

It's one data point - not out of 12,500 trials, but one trial (two counting the next day). The trials began the first time Mike noticed the spider.

Assuming that as the trials increase with no further occurrences of spiders, (say maybe two months = approx 60 trials), Mike will be checking less often for spiders. Or, it may be that the cost of checking is so low that he will continue to do so, but I would guess that his anxiety over the possibility of finding a spider would decrease over time.

It's natural and intelligent to follow the results of one data point given one trial. In fact, this is the argument for racial diversity, because if you have less experiences with one race, the more your are influenced by that small sample size. And that is a natural reaction. You need more "trials" to reach meaningful conclusions.

Sam G

In response to #17 and the comment about air travel being safer than driving.

This is an oft-repeated bit of conventional wisdom that I find myself unsatisfied with. I've only ever seen the comparison being made by expressing a ratio of deaths per mile traveled. However, I would argue that a more meaningful comparison would be a ratio of deaths per minute (or time unit) of travel. The distance is thus negated and what we're left with more accurately reflects how dangerous an activity is relative to the time spent engaging in that activity.

I'd guess flying would still be much safer, but not as it has so dramatically been shown.

Kevin H

As far as science goes, single data points are fine for generating hypothesis, just not for testing most hypothesis.

There are hypothesis, such as 'can plutonium undergo explosive nuclear fission' that have such dramatic outcome, that you can pretty safely call it on one successful trial. Really, it's all about signal to noise ratios. If you can have some estimation of the noise (plutonium just sitting there doesn't every spontaneously explode) then a single trial can be statistically significant.

Also, knowledge can be transformational, even with an N of one. The fact that the shoe bomber story was publicized means that a lot more people would get the idea that the shoe was a good place to hide contraband.

Joe

We build much of our airport security on n=1 data. I have to take off my shoes every time I go to the airport because one person tried to put a bomb in his shoe.

Ed

An “n of 1? finding is far superior to an “n of zero? finding. The latter seems often to be the approach of child experts, and "experts" in a great many fields. The professionals who used the evidence of the Baby Emily tapes were far more credible than their colleagues who used no evidence at all.

Is studying Tiger to understand how children learn golf really worse than not studying anyone at all? Do we really prefer to believe the views of an expert (who hasn't actually studied any cases of the subject under review), simply because he's an expert who says his pronouncements must be correct because he's an expert who knows more about the subject than anybody else and so his views on an unstudied aspect of the subject are more likely to be right?

Joe Gibson

What about legislation? It seems like any time an individual makes the national news, there is always some follow up legislation. The VA Tech shootings prompting a new gun control bill and an officer being struck by a car on a highway during a traffic stop leading to discussion of a "move- over" law in Massachusetts are some more recent examples.

Manuel

Re: #15, 16, db&5cott: When I was five, my father was stung by a scorpion in his towel. I have showered/bathed in more than one hundred cities around the world, lived in four different countries and six states, including places that do not have scorpions, and 39 years later I am still checking my towel by shaking it before I use it. Thus, I interpret my N=1 to not require an actual sting to learn to be wary of scorpions in my towel, and it seems irrational in a place where I know and recognize that scorpions do not live. Perhaps it is more of a habit now, like combing my hair the same way or any other behavior that might have once been great cause for reflection and consternation and then became an automatic custom.

sunshine

This isn't an example of "N=1" because "N=1" connotes a scientist deliberating in an ivory tower, which, I hope, is not the way Mike lives his day-to-day life (my sympathy to his family and friends if it is).

The appropriate term for what Mike describes is "single-trial learning." Single-trial learning is an obviously adaptive ability for organisms to have as they navigate the real world.

Alex

Don't sell n=1 research short. It is particularly good for some things, and bad for others.

When n=1 (or some other VERY low number), the researcher/team can look in depth, examining processes and interactions and potential causalities that larger studies cannot.

Yes, n=10,000 is great for some things, but it is lousy when trying to understand the mechanism of a phenomena. n=1 allows you to build a theory in ways that n=large number does not.

Of course, the test of any theory is its ability to predict. So, having built a theory, the researcher/team should look to test its predictions, either with n=small number studies or n=large number studies.

Steve

Also in this case even though the probability of a spider being there is extremely low (but not zero) given the fact that the cost of the spider being there (the yech factor, a spider bite) is relatively high AND the cost of checking is also extremely low (in terms of time and effort) it makes sense to check "just in case." This is true for most low probability high cost events where the cost of checking/prevention is low. Now if it took 2 hours and 5 different steps to check for spiders then this would obviously not hold.

DQKennard

He only *found* one spider in those 12,000 showers. That doesn't mean he was spider-free (though it could), only that he was knowledge-of-spiders-free.

Bernie

A related phenomenon is what behaviorists term intermittent positive reinforcement. A child who whines "I want a cookie" and finally gets one after dozens of 'trials' learns that whining is a valid path to attaining a desired end. And our towel-shaker will never stop shaking towels if just once a shake dislodges an unseen spider.