You've probably heard by now that the NTSB has recommended that states forbid drivers to use cell phones, whether hands-free or not. Here is a good AP article by Joan Lowy about what is known and not known about phone risk. She makes the excellent point that it's harder to argue for a ban when highway fatalities keep falling -- but that a falling death rate hardly means that cell phone use isn't dangerous. (Off-topic but not too dissimilar: Americans are losing their taste for the death penalty, theoretically because it's sometimes applied so haphazardly -- but in truth it's a lot easier to argue against the death penalty when the murder rate has fallen as dramatically as it has.)
In the AP article, Marcel Just of the Center for Cognitive Brain Imaging at Carnegie Mellon, puts in words why phones may cause a particular risk of distraction:
"When someone is speaking your native language, you can't will yourself to not hear and process it. It just goes in," Just said. Even if a driver tries to ignore the words, scientists "can see activation in the auditory cortex, in the language areas (of the brain). "
This would also explain why hearing someone else's cell-phone chatter in public is more annoying than it ought to be.