Levitt replies to the critical letter published in NY Times today

In the “Letters” section of today’s New York Times Sunday magazine, a letter by two doctors at the Children’s Hospital of Philadelphia criticizes our piece on car seats vs seat belts:

As pediatricians, scientists and leaders of the world’s largest study on children in crashes, we think that overinterpretation of findings from a single source of data led Stephen J. Dubner and Steven D. Levitt (July 10) to claim that child safety seats are no more effective than seat belts for 2- to 6-year-olds. They examined children in fatal crashes (about 1,200 per year) while ignoring the equally informative data on those in nonfatal crashes (450,000 per year). Our research, which includes over 25,000 in-depth interviews and over 800 crash investigations, consistently shows that child safety seats and booster seats significantly lower the risk of serious injury compared to seat belts alone. Their conclusions stand in stark contrast to the existing body of scientific data that support current child restraint recommendations, and are, in our opinion, irresponsible and dangerous.

Learn the facts at www.chop.edu/carseat. We hope that this misleading article does not cost a child his life.

Dennis Durbin, M.D., and Flaura Winston, M.D.
The Children’s Hospital of Philadelphia
Philadelphia

I may be wrong about car seats, but definitely not for the reason they claim (i.e. that we only look at fatal crashes). I have looked at three other data sets (one with all reported crashes in New Jersey, another with all reported crashes in Wisconsin, and one that is a nationally representative sample of all crashes in the United States). All three of these data sets cover a full set of accidents, not just crashes with fatalities. And in all three data sets the results obtained are virtually identical to the results using the fatal crashes.

Moreover, even in the original paper which did focus on fatal crashes, I employ the techniques of Levitt and Porter (2001) which deal with sample selection (the primary concern one would have with using just fatal crashes) in an extremely convincing manner.

What I find particularly interesting is, despite the NY Times claim that they “fact check” letters to the Sunday magazine, and despite the fact that the NY Times was fully aware that I have analyzed these three additional data sets, they chose to run the letter with this false critique.

I’m not against the Children’s Hospital of Philadelphia folks getting the chance to respond, because I think they deserve that chance, but I find it disturbing that the NY Times would knowingly publish a letter with arguments it knows to be false. Further, I find it interesting that in their “response” on their web site, there is no mention whatsoever of any particular flaws in my analysis, or the issue of my focusing on fatal crashes alone. They simply rehash the prior studies that have been done and say our NY Times column might have “confused” parents.

The prior research, it should be noted, is heavily (though not exclusively) based on a sample that these same doctors have assembled from customers of State Farm Insurance, who the researchers call and ask questions a few days after the accident. While this has been a huge and admirable endeavor on their part, the methodology is not without limitations:

1) A substantial fraction of the State Farm clients decline to respond to the researcher inquiries. No information is provided about how these non-respondents differ from respondents.

2) One could imagine that there are incentives for the respondents to lie about the type of restraint they had their child in. What would you say if you had been driving with your child unrestrained, the child had been injured, and your insurance company called you on the phone a few days later to inquire about details of the crash? If there was no car seat in the vehicle, it would be hard to lie and say you had the child in a car seat since there are police reports. But, you may think that you can get away with saying your child was wearing a seat belt, since all cars have seat belts.

3) If you asked social scientists whether they’d rather have extensive data collected at the crash scene for some universe of crashes, or rely on parent surveys a few days later for your data, I believe that their answer would almost unanimously be that they prefer the universe of crash scene data. Which is precisely the data I am relying on.

About 10 days ago I emailed the director of the team at Children’s Hospital of Philadelphia who do this research, telling her that I would share all my data and programs with them if they were interested, and requesting that they provide the same courtesty to me. She has not responded.

If indeed, the folks at Children’s Hospital of Philadelphia are truly interested in child safety, rather than just being seen as the leaders in child safety, I would hope that they would be willing to work together to figure out why these two sets of results differ so dramatically.