Levitt replies to the critical letter published in NY Times today

In the “Letters” section of today’s New York Times Sunday magazine, a letter by two doctors at the Children’s Hospital of Philadelphia criticizes our piece on car seats vs seat belts:

As pediatricians, scientists and leaders of the world’s largest study on children in crashes, we think that overinterpretation of findings from a single source of data led Stephen J. Dubner and Steven D. Levitt (July 10) to claim that child safety seats are no more effective than seat belts for 2- to 6-year-olds. They examined children in fatal crashes (about 1,200 per year) while ignoring the equally informative data on those in nonfatal crashes (450,000 per year). Our research, which includes over 25,000 in-depth interviews and over 800 crash investigations, consistently shows that child safety seats and booster seats significantly lower the risk of serious injury compared to seat belts alone. Their conclusions stand in stark contrast to the existing body of scientific data that support current child restraint recommendations, and are, in our opinion, irresponsible and dangerous.

Learn the facts at www.chop.edu/carseat. We hope that this misleading article does not cost a child his life.

Dennis Durbin, M.D., and Flaura Winston, M.D.
The Children’s Hospital of Philadelphia
Philadelphia

I may be wrong about car seats, but definitely not for the reason they claim (i.e. that we only look at fatal crashes). I have looked at three other data sets (one with all reported crashes in New Jersey, another with all reported crashes in Wisconsin, and one that is a nationally representative sample of all crashes in the United States). All three of these data sets cover a full set of accidents, not just crashes with fatalities. And in all three data sets the results obtained are virtually identical to the results using the fatal crashes.

Moreover, even in the original paper which did focus on fatal crashes, I employ the techniques of Levitt and Porter (2001) which deal with sample selection (the primary concern one would have with using just fatal crashes) in an extremely convincing manner.

What I find particularly interesting is, despite the NY Times claim that they “fact check” letters to the Sunday magazine, and despite the fact that the NY Times was fully aware that I have analyzed these three additional data sets, they chose to run the letter with this false critique.

I’m not against the Children’s Hospital of Philadelphia folks getting the chance to respond, because I think they deserve that chance, but I find it disturbing that the NY Times would knowingly publish a letter with arguments it knows to be false. Further, I find it interesting that in their “response” on their web site, there is no mention whatsoever of any particular flaws in my analysis, or the issue of my focusing on fatal crashes alone. They simply rehash the prior studies that have been done and say our NY Times column might have “confused” parents.

The prior research, it should be noted, is heavily (though not exclusively) based on a sample that these same doctors have assembled from customers of State Farm Insurance, who the researchers call and ask questions a few days after the accident. While this has been a huge and admirable endeavor on their part, the methodology is not without limitations:

1) A substantial fraction of the State Farm clients decline to respond to the researcher inquiries. No information is provided about how these non-respondents differ from respondents.

2) One could imagine that there are incentives for the respondents to lie about the type of restraint they had their child in. What would you say if you had been driving with your child unrestrained, the child had been injured, and your insurance company called you on the phone a few days later to inquire about details of the crash? If there was no car seat in the vehicle, it would be hard to lie and say you had the child in a car seat since there are police reports. But, you may think that you can get away with saying your child was wearing a seat belt, since all cars have seat belts.

3) If you asked social scientists whether they’d rather have extensive data collected at the crash scene for some universe of crashes, or rely on parent surveys a few days later for your data, I believe that their answer would almost unanimously be that they prefer the universe of crash scene data. Which is precisely the data I am relying on.

About 10 days ago I emailed the director of the team at Children’s Hospital of Philadelphia who do this research, telling her that I would share all my data and programs with them if they were interested, and requesting that they provide the same courtesty to me. She has not responded.

If indeed, the folks at Children’s Hospital of Philadelphia are truly interested in child safety, rather than just being seen as the leaders in child safety, I would hope that they would be willing to work together to figure out why these two sets of results differ so dramatically.


rini

People like to believe that they can protect their children more than is actually possible. To take away the magical amulet of the car seat is to rob people not only of preconceived notions, but false security. Truth? Schmuth.


(Oh, and don't forget the car seat manufacturers. They don't want this data out either.)

Rian

It is possible that the doctors simply don't want to entertain the idea that they might be wrong. This is not an uncommon phenomenon, in my experience; I work in the medical field, and there's no shortage of ego for many practitioners.

I'll be watching to see what their response is. It's sad that they responded in such an ignorant, knee-jerk fashion.

rini

I am in the medical field as well. Unfortunately, I agree with you. I try to be a counter-example.

Mateu

Shouldn't that be "Which are precisely the data I am relying on." ?

But the reason for the NY Times' behavior seems entirely different to me: fear. We have now been using car seats for many years in America, and have (I think) come to assume they are the safest option for children. Who wants to be the first to use their children to test that theory? Moreover, what major media outlet would want to expose itself to the liability lawsuit from the parent that puts her child in a car seat and has something bad happen?

adeu,
Mateu

HUICHIEH LOY

Mateu: "data" can take both singular and plural verbs--and the former when it is taken as a mass noun term, like "water" (after all, usually "little data/water", rather than "few data"). "Data" in the sense of "information" is a mass noun.

I'm disappointed that the NYT didn't bother to publish the reply--even with a suitable "this letter does not represent our views" disclaimer.

jay

"About 10 days ago I emailed the director of the team at Children's Hospital of Philadelphia.... She has not responded."
How do you know that your email got through to her, and was not swallowed by a spam filter?

Anonymous

Steve: I employ the techniques of Levitt and Porter (2001) which deal with sample selection (the primary concern one would have with using just fatal crashes) in an extremely convincing manner.

Yikes. Your analysis is convincing to you. Whoopee.

You and Steve have half a dozen children between you. You have suffered the loss of an infant. Yet you are petulantly pushing your own view here when you think there's no difference between car seats and seat belts and Children's Hospital thinks car seats are better.

If you are right but parents believe them, people who buy car seats for toddlers are wasting money. If they are right but parents believe you, children will die.

There is an "asymmetrical loss function." A person who believes you if you are wrong is in deeper trouble than a person who believes them if they are wrong.

You appear to lack a basic understanding of economics 101.

Read more...

Steven D. Levitt

To anonymous who says I don't understand econ 101:

You totally miss the point. The goal is to figure out whether car seats work or not. There is an asymmetric loss function only because there is uncertainty about the true parameters. If we knew the true effect of car seats, then we could act accordingly. The point of my research is to learn these parameters.

If car seats don't work, the answer probably wouldn't be to throw them away and go merrily on our way. More likely, it would be to take the resources spent on car seats and invest them in something that provides much larger benefits.

There are many things I lack a basic understanding of. I guarantee you that econ 101 is not one of them.

Steve Levitt

qualityg says

Steve,

How about contacting those myth busters on cable TV and ask them to test both methods (make sure they are not affiliated State Farm).

Data + Information is OK but add Application and you are getting closer to obtaining knowledge.

D+I+A = K

Crazy

If the doctors are truly well intentioned they ought to exchange data with Levitt.

We already know the details of Levitt's approach/data but we really don't have a clue on the details of the doctor's survey data.

Andy_Martini

At 7:39pm, Anonymous wrote, "If you are right but parents believe them, people who buy car seats for toddlers are wasting money. If they are right but parents believe you, children will die."

You are making the assumption that Dr. Levitt's analysis is wrong. Yet, what if his analysis is correct? For that matter, what if the data sets show that BOTH forms of restraint save lives? That has yet to be proven, and even Dr. Levitt has admitted that may be the case.

Part of econ 101 is asking the questions from all sides. Another basic tenet of econ 101 is understanding it is NOT an exact science. I'm sure there is a university in your area offering refresher courses.

And that was pretty insensitive to bring up his loss considering it was not a loss due to a car accident as you may have learned in his book.

Noumenon

Good defense of your position, Steve.

Praiseworthy defense of Steve, andy_martini.

Phil

Dr. Levitt,

Please do let us know if the Children's Hospital of Philadelphia replies and shares their data.

Is it normal academic courtesy to routinely share one's data when asked, or is it an unusual request that is legitimate to refuse?

miss rogue

Looks like Seth Godin is skeptical:

http://sethgodin.typepad.com/seths_blog/2005/07/the_other_side_.html

...does he syndicate your blog? Maybe you should comment there. He has a big following (besides the two of you have to speak together in Toronto in the fall).

josh

anonymous: if you're going to say something stupid in a post, at least use your name.

The fact is Levitt isn't 'petulantly pushing' his own view, as you so articulately alliterated, but rather trying to find out what is actually happening. I think this will save more lives in the end whether his findings are right or wrong.

Anonymous

http://www.chop.edu/consumer/jsp/division/generic.jsp?id=77974

Your child should use a booster seat until he is at least 8-years-old, unless at least 4'9" tall.

--
I think both sides agree that:

a. it's good (safety is increased) to supplement a vehicle's Passenger Restrait System for passengers under the age of 2.

b. it's bad (safety is decreased) to supplement a vehicle's PRS for passengers over the age of 10.

The controversy is where the cutoff is. It's not just a matter of wasting money. Booster seats probably have a higher risk of ejection from vehicle and other downsides. So at a certain age, the risks are greater than the benefits.

It's a good bet that state legislators overshot the optimal age due to pressure from car seat manufacturers and their cronies in Congress.

DF

Michael O.

It is my impression from reading several medical papers that, basically, (many) doctors and medical researchers have extremely strong priors on what is good for you, and get really excited about data that supports their conclusion and hostile to everything else. This is probably common everywhere, but what's troubling is that in medicine all the researchers seem to be on the same side of many issues.

Here is my favorite gem, on smoking, published in BMJ --- as far as I know, the medical equivalent of Econometrica or AER:
http://bmj.bmjjournals.com/cgi/content/abstract/bmj.38055.715683.55v1

It would probably be useful to collect the issues on which almost all medical profession is on the same side, put the list on a website, and then have economics graduate students debunk them one by one as their second-year papers.

Crazy

Yes it's pretty suspicious to not share data when the findings are challenged unless the the authors were bound by a NDA with regards to the data. Levitt shared the abortion data with plenty of critics.

Jim Voigt

I tried to make the point earlier that I believed that the average person reading this article would understand it to mean that car seats are not effective and are therefore not necessary. I attempted to explain that this might be the perception, EVEN THOUGH it is NOT the message Levitt is actually trying to send.

I raised the possibility of this misunderstanding, especially among people who do not hold a strong understanding of economics, and asked the question whether it was responsible to print such an article when that dangerous a misperception is the potential outcome. From the comments here and letters to NYT, it appears that such a misperception is not only a possibility, but a reality.

The response I received was a technical explanation for why people would not misunderstand. In other words, a defense of the data. This struck me as odd since I was asserting that the lack of technical knowledge would be the catalyst, and not the solution for just such a misunderstanding. I am not saying the data is wrong, I am saying that people might very well misunderstand the data as it was interpreted. The authors appear to deny that this is a risk whatsoever, because their only responses are to continually defend the data.

Essentially, the fine authors of this article seem determined to stick by their data, which is fine. I believe it is not fine, though, for them to continue to deny that there is a strong possibility that people could easily misread their article and discontinue use of a car seat.

So, what is the extraodinary benefit of this article that is sufficient to outweigh such a risk? In other words, what exactly are the authors trying to accomplish here?

Read more...

Anonymous

Since most states have child-restraint laws that specify the size and weight of children that must use car seats or boosters, I would expect the number of car-seat reports to be higher for both fatal and non-fatal crashes. But there is a monkey ranch with the car seats as well--your point concernin lying about belts is valid, but it pales in comparison with incorrectly fastened car seats or children in car seats that are not properly restrained. Just from a small circle of acquaintances, I can recall a substantial number of instances where the seats were not properly fastened--either the belt had been unbuckled, or the tether was attached to incorrect receptacle or did not fit, or the seat was simply loosely fastened. The number of instances of parents simpy plopping the kid in a seat without buckling at all point, or an older kid unbuckling himself is quite troubling as well.

This means that the actual results are somewhat overstated on both sides. I would even venture to say that the numbers are more overstated on car seats, where there many more ways to have the restraint go wrong (either accidentally or willfully), resulting in a bias against car seats. Your "methodology limitations" (2) and (3) fail to offer a comprehensive analysis, making it appear that data are biased against seat belts.

One could argue that we should consider all data--including both correct and faulty applications--since it is the object and not the application of the object that's considered. But this argument needs to be made to be considered--you simply failed to account for it. Furthermore, even if we consider all such events, there is still a good reason to keep track of different instances, if possible. The data may point (I have not seen the details, so I can't say for sure) that correcly applied, car seat have a tremendous advantage over correctly applied seat belts, so the campaign for car seats should have two components--first, to encourage their use, second, to make sure that they are used correctly.

This is a standard approach in public health policy--advocate for correct use of a safety device rather than simply argue that the device itself makes one safer. The presence of seat belts in a car does not make one safer, but weating them might. An incorrectly worn/sized helmet will not protect one in a bike crash, but this is not a reason not to wear a helmet. From this perspective, your methodology is as flawed as your opponents', if not more so.

Questioning publi-health policy precepts is a good idea, but it is not clear where the presumptive burden should lie. For libertarians, the presumption is on the side of avoidance of regulations--so the assumption is always that we should not advocate the use of safety devices unless there is clear incontrovertible proof of their effectiveness. The more liberal interpretation is to advocate the use of safety devices if the data supports that they are at least as effective and may be more effective than non-use. Of course, even beyond positive proof, liberatarian would steel oppose any regulations, claiming that it should be individual choice whether a safety device should be used or not (e.g., seat belts, motorcycle helmets--in a slightly different way, this also applies to tobacco use). The trouble is, of course, in the fact that the libertarian approach only considers individual impact and not the social impact of increased safety as a result of the regulations. From what I have seen, I suspect your sympathies to lie on the libertarian side.

Read more...