How Valid Are T.V. Weather Forecasts?

Eggleston and daughterEggleston and his daughter two minutes before it began to hail. Says Eggleston, “Hail was not in the forecast.”

A gentleman named J.D. Eggleston recently wrote to us with a rather interesting report, a nice piece of D.I.Y. Freakonomics concerning the accuracy of local T.V. weather forecasts. I thought it was interesting enough to post in its entirety here on the blog, and I hope you agree. Before we get to the report itself, here is a little background information from Eggleston himself:

I live with my wife and two kids, 15 and 12, in rural northwest Missouri. I earned a bachelor’s in electronics engineering technology from DeVry University in 1987. I’ve been an electronics engineer, software engineer, and for the past 13 years I’ve owned and operated a consumer electronics retail business.

I’ve always loved math and statistics and the information that can be learned from studying them. Growing up, I was told that people like me were called “weird.” Since reading Freakonomics, I now know they are called “economists.” It’s good to know I’m not alone.

The forecasting study began in April of 2007 when my fifth-grade daughter was given a school assignment to monitor the temperature and rainfall at our home for a week. Our family members are big T.V. watchers, and our house is loaded with the latest D.V.R. (digital video recorder) technology. So we decided to document not only the weather results at our home, but also to record the 10 p.m. newscasts for channels 4, 5, 9, and 41 and compare our home results to those reported by the Kansas City T.V. stations.

And while we were at it, we decided to also document each station’s weather predictions and compare them to the actual results to see if one station was better than the others. For a non-T.V. weather source, we also recorded the predictions of the federal government’s National Weather Service each evening.

And now for the report. The takeaway message? Do not plan your weekend activities based on the T.V. weather forecasts unless it is already Thursday — but waiting until Friday would be even better.

How Valid Are T.V. Weather Forecasts?
A Guest Post
By J.D. Eggleston

The authors of Freakonomics posed the question, “Do real estate agents have your best interests at heart?” Then they statistically showed they (the real estate agents) do not. So what about meteorologists? How accurate are their forecasts? Do they even care?

A seven-month study of weather forecasting at Kansas City television stations was conducted over 220 days, from April 22 to November 21, 2007. The seven-day forecasts for both high temperature and P.O.P. (probability of precipitation) for each station’s 10 p.m. telecast and from the N.O.A.A. Web site were recorded. For stations that did not offer a P.O.P. in the form of percent likelihood, the best impression of percent likelihood that could be inferred from the meteorologists’ words and graphics were used. The results of Kansas City’s high temperature and rainfall as reported at the K.C.I. airport weather station — which are the data that become the official record for weather at Kansas City — were also recorded. Those results were then compared to the high temperature and P.O.P. predictions to determine forecasting accuracy for each source for each of the seven days predicted.

The results were quite enlightening, as were some of the comments of the local meteorologists and their station managers. Here a few of the quotes we received:

“We have no idea what’s going to happen [in the weather] beyond three days out.”

“There’s not an evaluation of accuracy in hiring meteorologists. Presentation takes precedence over accuracy.”

“All that viewers care about is the next day. Accuracy is not a big deal to viewers.”

Temperature

INSERT DESCRIPTION

All of the chief meteorologists were asked, “How close does your high-temperature prediction have to be to the actual temperature for you to feel like you did a good job?”

Without exception, all of the meteorologists answered, “within three degrees.”

The chart above shows the results of the stations’ temperature prediction accuracy for their full seven-day forecasts. For next day predicting (one day out), all stations met their “within three degrees” goal. For two days out, all but one was within three degrees. But for three days out and beyond, none of the forecasters met their three-degree benchmark, and in fact get linearly worse each day.

The conclusion to be drawn here is not so much that one station is better than another, since all of them seem to be similar in accuracy — and most people won’t alter their plans based on a couple degrees of temperature. Rather, all of our stations did not do a good job by their own definition of plus/minus three degrees beyond two days out.


Getting It Right the First Time

INSERT DESCRIPTION

When we get our first predictions for, say, June 13th, it will be the seventh day of a seven-day forecast made on June 6th. The following day, it will be the sixth day out, then the fifth, then fourth, and so on until it is tomorrow’s forecast.

Have you ever noticed that the prediction for a particular day keeps changing from day to day, sometimes by quite a bit? The graph above shows how much the different stations change their minds about their own forecasts over a seven-day period.

On average, N.O.A.A. is the most consistent, but even they change their mind by more than six degrees and 23 percent likelihood of precipitation over a seven-day span.

The Kansas City television meteorologists will change their mind from 6.8 to nearly nine degrees in temperature and 30 percent to 57 percent in precipitation, showing a distinct lack of confidence in their initial predictions as time goes on.

The prize for the single most inconsistent forecast goes to Channel 5’s Devon Lucie who on Sunday, September 30th predicted a high temperature of 53 degrees for October 7th, and seven days later changed it to 84 degrees — a difference of 31 degrees! It turned out to be 81 that day.

A close second was Channel 4’s Mike Thompson‘s initial prediction of 83 for October 15th, which he changed to 53 just two days later. It turned out to be 64 on the 15th.

Even more conclusively than the temperature accuracy graph, this prediction variance graph shows that 21st century meteorology is not developed enough to provide a week of accurate temperature forecasting.

Meteorologists take a blind stab at what the high temperature and rain possibilities might be seven days out, and then adjust their predictions on the fly as the week goes on. As mentioned earlier, one meteorologist told us: “We have no idea what’s going to happen beyond three days out.”

Will It Rain?

Precipitation will affect the average person’s plans more significantly than temperature. We rely on meteorologists to be accurate in their rainfall predictions so we can plan the events of our lives. Parades, gardening, ball games, outdoor work, car washing, construction work and farming are all affected — positively or negatively — by rain.

We could just assume it will not rain, but it would be nice to have a little heads-up. In measuring precipitation accuracy, the study assumed that if a forecaster predicted a 50 percent or higher chance of precipitation, they were saying it was more likely to rain than not. Less than 50 percent meant it was more likely to not rain.

That prediction was then compared to whether or not it actually did rain, where “rain” is defined as one-tenth of an inch or more of rainfall reported at K.C.I. Anything less than that is so irrelevant, it would likely make no difference in people’s lives.

INSERT DESCRIPTION

The graph above shows that stations get their precipitation predictions correct about 85 percent of the time one day out and decline to about 73 percent seven days out.

On the surface, that would not seem too bad. But consider that if a meteorologist always predicted that it would never rain, they would be right 86.3 percent of the time. So if a viewer was looking for more certainty than just assuming it will not rain, a successful meteorologist would have to be better than 86.3 percent. Three of the forecasters were about 87 percent at one day out — a hair over the threshold for success.

Other than that, no forecaster is ever better than just assuming it won’t rain. If you think that’s bad, sadly it gets worse:

The data for the precipitation accuracy graph was taken from all days of the study. For many of those summer days it was clearly obvious there would be no rain, and thus those days were no challenge for the meteorologists. A better measure of a forecaster’s skill would be to exclude the days when there was clearly no chance of rain. After all, if you wanted to measure a golfer’s putting skill, you would not have him putt his test putts from only six inches away from the cup. You would want to challenge him with putts from five to fifteen feet — putts that could readily be made or missed.

For that type of meteorologist test, we only included the days that it either rained or the meteorologist predicted it would rain, thus eliminating the days where it clearly was not going to rain. The following graph shows the results.

INSERT DESCRIPTION

Because conditions for rain on these days were more likely and more challenging to predict, we lowered our benchmark for success on this test from 86.3 percent to 50 percent. Sadly, four of the five stations topped the 50 percent goal only on their next-day forecast.

For all days beyond the next day out, viewers would be better off flipping a coin to predict rainfall than trusting the stations on days where rain was possible. Oddly, N.O.A.A. — which had been one of the better forecasters in our other evaluations — was the worst in this one, especially when predicting three days out and beyond.

When N.O.A.A. meteorologist Noelle Runyan was questioned about this, she stated, “Our forecasts are more conservative than the television stations. We raise our P.O.P. predictions to over 50 percent only when we are sure of rain.” This statement and the data above are another illustration of how — with the data and tools given to them — today’s meteorologists cannot confidently predict the weather beyond three days out.

Second Fiddles

Have you ever wondered if the forecast you get from the weekend meteorologist or vacation replacement is as good as from the chief meteorologist? Many people do, so on July 5th, comparisons of the accuracy of each station’s chief meteorologist to their weekend replacements were made.

Because this comparison did not start to be made until July 5th, the numbers shown in the table below may not match the numbers published for station-to-station comparison in other parts of this report. For each of the lines below, the top name is a station’s chief meteorologist, and the second line is their back up. Here is how the individual meteorologists fared.

INSERT DESCRIPTION

At Channel 4, Mike Thompson’s weekend man is Joe Lauria. From the table above, we can see that Lauria is actually much better than Thompson in temperature accuracy from about .5 to 2.5 degrees better across the seven-day range. Regarding precipitation, Thompson is slightly better than Lauria one or two days out, but Lauria is more accurate three to seven days out, and on the challenging days.

At Channel 5, Katie Horner‘s weekend replacement is Devon Lucie. As with Channel 4, it appears Channel 5’s weekend forecasts are more accurate for both temperature and precipitation, but only slightly.

At Channel 9, Pete Grigsby is the weekend man for Bryan Busby. Here, Busby is better at precipitation and at one to three days out on temperature. Grigsby is better four to seven days out on temperature.

Channel 41’s weekend weatherman is Jeremy Nelson. When it comes to temperature, Nelson is not as good as Lezak one or two days out, but is better than Lezak longer range. For precipitation, both are pretty even.

The New and Improved Weather

Back in the 1990’s in an episode of the television show L.A. Law, a nerdy but effective meteorologist sued his former employer because they fired him and hired a comedian to do the weather. While none of Kansas City’s meteorologists are uneducated, stand-up comics, there does seem to be an unfortunate emphasis of style over substance.

When station managers were asked about this, one said, “There’s not an evaluation of accuracy in hiring meteorologists. Presentation takes precedence over accuracy.” And when discussing accuracy (or the lack thereof) of a seven-day forecast, another station manager stated, “All viewers care about is the next day. Accuracy is not a big deal to viewers.”

When weather events occur that really are news — flooding, tornadoes, ice storms — all of the Kansas City meteorologists do an excellent job of informing their viewers, as do most forecasters across the country. Likewise, the stations allow their meteorologists ample time to report these serious weather events, be it in their 5, 6, or 10 p.m. telecasts, or by interrupting regular programming when necessary.

One of the two major weaknesses in television meteorology today is the “non-event” days — the boring, run-of-the-mill days when no significant weather events are upcoming. It is unfortunate that 13 percent of each news telecast (actually about 20 percent if you discount the commercials) is dedicated to a weather forecast that is mostly time-consuming fluff.

The meat of such forecasts could easily be condensed to one minute or less, or maybe even a crawl at the bottom of the screen that runs for the full telecast. Reduction of the weather segment on days when there is no weather news would allow for more thorough reporting of world, national, and local news.

The other major weakness is that ratings drive television. Sadly, the data show that stations are so consumed with ratings that accuracy in weather predictions takes an irrelevant back seat to snappy patter and charm. When directly asked if accuracy mattered in forecasting, every station manager and meteorologist said it did. But when asked what steps they had taken to measure and ensure accuracy, they were without answers.

No meteorologist or television station kept records of what they predicted, nor compared their predictions to actual results over a long term. No meteorologist posts their accuracy statistics on their résumé. No station managers use accuracy statistics in the hiring or evaluation of their meteorologists.

Instead, the focus is on charm, charisma, and presentation. Their words say they care about accuracy, but their actions say they do not. Yet, they wish to continue providing inaccurate seven-day forecasts that are no more than a semi-educated shot in the dark because a) their competitors do and b) they can get away with it since they think the public does not know how inaccurate they are.

Until the public demands change in the form of lost ratings from this hollow practice of “placebo forecasting,” T.V. weather forecasts will continue to blow smoke up our … upper-level-lows.

Until this change comes to pass, we must take what we see on T.V. with a grain (or perhaps block) of salt. And if you really want to know what weather will occur in Kansas City tomorrow, find out what happened in Denver today.


teneriff

The weather segment of the local news is THE MOST PROFITABLE part of the broadcasting day.

As George Stephanopoulos and Charles Gibson said of the last debate: It's entertainment!

The Weather Channel is for sale - minimum bid: $1 billion. THATS'S ENTERTAINMENT!

Jared C

The simplification of the weathre forecast into "rain" or "no rain" seems to be a mistake - the weatherman is not claiming that the weather is deterministic. As an analogy, if I claim there is a 1/6 chance a rolled die will come out a 2, and it comes out a 2, one would not say I was wrong.
In the case of the weatherman, consider a series of predictions that there is a 60% chance of rain the next day. If the weatherman is exactly correct, treating all of these as a prediction for rain will results in finding him incorrect 40% of the time. A better analysis would look at the actual occurance rate for rain given the forecast probability. For instance, compare the actual percentage of days it rained when the forecaster claimed a 75% chance of raining.

Claudia

The seven-day forecast has real financial implications for businesses, given that viewers expect the forecasts to be somewhat accurate. A few years back I read an article about how Bed & Breakfast owners from Cape Cod, MA, were losing money because people would cancel reservations based on the 10-day forecast, but often the forecast was incorrect. It's rather irresponsible to make such forecasts when they are so inaccurate, because the general public assumes they are somewhat reliable.

XCSkierDude

Too bad about the rest of you guys. Here in Chicago we have Tom Skilling, who actually does care about the quality of his forecasts; he even cares about whether his viewers and readers understand the science behind the forecasts. Maybe you can catch a glimpse of him sometime when you're stuck at O'Hare next winter, in a snowstorm the KC weatherfolk missed.

Richard King

I can remember when an elementary school class was charged to count the cherries in a McDonalds cherry pie product as part of a statistics assignment. They found that in several "pies" ther were none! This caused McDonalds to immediately change their recipe. Perhaps this wonderful weather study can have the same affect.

salliek76

My father is a farmer who needs to plan his business around the forecast, specifically with regard to precipitation. A wrong decision on when/whether to cut hay, rake it, bale it, move loads of feed or fertilizer, etc., could cost him thousands of dollars. In a fit of frustration many years ago, he began a journal where he recorded the predicted weather and the actual weather. I think he ended up sending it to the local TV station after gathering about a month's data, and they sent him an umbrella with their logo. Not long after that, they started doing a contest where they'd draw somebody's name and give them a free umbrella if that day's forecast was wrong; I can't remember exactly how "wrong" was defined. It always seemed to me that they were creating more publicity by getting the forecast wrong than by getting it right, though.

mukul kantharia

Weather forecast, if it is simply a revenue earning tool, then it will never reach a level of accuracy we want. I believe, as the TV Stations cover large geographic areas, they essentially have no way to be accurate for the covered areas. When I lived in Queens, New York, there was a good chance weather forecast will be some what accurate. Since I have moved to Columbia, Maryland and the TV Station are mostly covering DC, VA areas or Baltimore areas our forecast is generally way off. I do not know how they can do better.

Bachelor

The more people learn and understand weather, the less they complain and the more they appreciate the weather forecasts. Too many people equate a bad forecast with a lazy forecaster. It doesn't work like that! Many of you will never understand why unless you walk in their shoes for a day.

Amina

TV has always been about charisma and presentation rather than accuracy and reliabilty.

Brett

What a coincidence that I was reading James Gleick's book on 'Chaos' last night and then I come across this question about the validity of TV weather forecasts. After reading about Edward Lorenz's discovery of the 'butterfly effect' and non-linear systems trying to model weather, I now see any long range forecasting as a folly. Worst yet, it's really a sham perpetuated on the public to fill air time on the local news.

Hence, even though I admire your statistics gathering, your results will have little to no effect on predicting which weather forecaster can better predict the weather. The answer is that a nonlinear system will not present accurate answer and it's those nonlinear computer models that these reporters work off of.

Put in a more sophisticated way, Laplacian Newtonian Determinism is a myth. I'd love to take credit for that discovery, but I think Edward Norton Lorenz realized that first.

Having spent my youth producing econometrics reports for investment bankers, I can tell you that they just wanted someone to tell them a story on paper. Like the weather, there are many nonlinear irregularities in human behavior that economists simply cannot model. It didn't matter how far off from truth our reports were. Predicting oil prices out 10 years was the norm for us. Unfortunately, we were lucky to get in the ball park a month or two out. Usually that is luck, but by the time it happened the oil deal or generator had been built. There are alot of witch doctors in modern America. Just start looking.

Still, intuition does seem to garner a much better weather predicting accuracy than NOAA's computers as you pointed out. Perhaps that's not just the luck of saying it won't rain every day. Perhaps the billions of hairs in our noses and skin, combined with our other senses drive the tiny quantum energy of the brain into offering a solution to the Butterfly Effect - a solution only a quantum computer could provide. Then again, maybe it's just a lucky guess :-)

Brett, Arizona
Dry today and for the rest of the week.

Read more...

Andrew

As an amateur meteorologist, I can tell you that much of the problems observed in this report are not a failure of meteorology but a failure of presentation. These failures of presentation have been compounded by the author's methods which are valid but don't necessarily convey a lot of information.

Part of the problem is the need to present quickly, graphically, and in a way that holds the viewers attention and can be quickly and easily understood by the lowest common denominator or those with a bare minimum of interest.

You'll also note NOAA did better than local mets. This is because NOAA employees actually have to be accurate and well trained. Most TV mets just copy off the NOAA information and ad their own flavor. It's also usually out of date.

Probability of precipitation is not really a useful statistic in many circumstances. For example saying 30% chance of precipitation today, doesn't convey the same information as "we are about to be hit by a major rainstorm and I think there is a 30% chance the rain will start before 7pm."

If you're interested in understanding conceptually what is happening and the possibilities of what might happen reading NWS "Area Forecast Discussions" or AFDs will convey infinitely more information in a few short paragraphs than a TV met does in his entire presentation.

The way the research was graded is not that informative either.

The way temperature was graded is informative.. 2 degrees off for today.. 6 degrees off for 7 days away. That's actually quite good. The temperature could be 2 degrees different on the other side of the street, never mind across town. And 6 degrees for seven days away tells me about what temperature it will be.. cold/cool/warm/hot.

Also the precipitation grading methods are very flawed. The author has graded how often it rains when the met says the probability is > 50%. Of course it will only rain ~80% of the time when the met says >50%. The author says the met was wrong when he says 60% chance of rain and it doesn't rain. The met wasn't wrong.. it was that other 40%. When SHOULD be graded is how often it rains when the met says 60%.. if it rains 60% of the time when he says that, the met is perfect. The author is asking "does it always rain when the met says there is a >50% chance of rain?"

Of course not.

Also he is counting only rainfalls above .10". That's not fair considering the met may have orally conveyed there is a 60% chance for a short light rain shower between 6 and 8pm. If it rained .08" in at 7 -8 pm.. the met was correct.

There are much more accurate and scientific ways of grading meteorology. It's most easy to grade a forecast model. Forecast models have been steadily improving since their invention. For the most up to date, comprehensive understanding of meteorology and of your local forecast.. you should look at computer model output. Do a google search for modern computer models such as the GFS, ECMWF, NAM, GEM, or UKMET. This is the data that mets look at.. and then make a forecast from. By the time it gets to you it is usually 12-36 hours old. First it has to be processed by NOAA (since most of the met field relies on them to some extent). NOAA offices only update fully twice a day. When they first release their report the data they are using is often about 6 hours old.. then it is eventually picked up by local mets..and by the time it gets into their forecasts it is usually at least 12 hours old. The weather channel is notorious for not updating with recent data. So part of the problem is a 1 day forecast.. is usually using 2 day old data!

Read more...

Eric

In Australia, the media get all their forecast from the Bureau of Meteorology, so the idea of different stations giving their own prediction is a novel one. Most TV channels add other information, such as good places for fishing/surfing, locations of speed cameras (so you can break the speed limit anywhere else??), fuel prices (which often follow a weekly cycle with 2x-amplitude > 10%).

One thing the forecasts seem to lack is any measure of variance, eg the 7-day forecast could put a ? next to the number for any day where some % of the probability distribution does not fit within a number of degrees of the main number given. There must be some days when they know their temp 4 days off will be close, and some where they know it could go anywhere.

Andrew

Brian L, post #176,

You're actually wrong on both counts. POP is not the percent of surface area receiving precip in a defined area. For example, it could be that there is a 60% chance all or most of the area will get precip and a 40% chance no-one will. Maybe the cold front wasn't quite as strong as expected, and thunderstorms failed to develop on it.

Also, your point about computer models is wrong too. Often models do just the opposite. In medium term projections (3-10 days away) the models often show drastic predictions and trend away from long-term averages. If you're familiar with meteorology you would know that the GFS model almost always shows a fantasy storm of the century blizzard at the end of its prediction cycle. This is because one little flaw in the model, or in the in-put data becomes magnified the farther out the prediction goes. The model doesn't use long-term averages in its predictions, it uses raw data and calculations.

Read more...

Christopher Bowns

From #5: "If meterologists can't predict the weather more than three days in advance, why is the “Global Warming Gang” so sure they can predict the weather 20 years in advance?"

Because meteorology is different than climatology.

sck

I live in the Silicon Valley and have to tell you I wonder all the time why the weathercasters at all the stations, major networks and a couple SF local stations, do not seem to get the weather forecast right, most of the time. The funny thing is they will all predict the same thing, and it will not correct. The incredible thing is they are using technology. I actually think that back when meterologists did their own weather predictions, they were more accurate. I'd love to hear from one of the weather casters in the bay area to hear what they have to say. Glad you did this story.

Brian L

There are two points about weather statistics on which I would appreciate some clarification. One, my understanding, and of some previous posters, is that the probability of precipitation is a forecast of the surface area receiving precipitation in a defined area.

Second, I assume no matter how sophisticated and/or unbiased a weather model might be, it would trend toward long-term averages in medium-term projections. The trending would make even honest forecasters look worse over time.

Alan Gunn

Consider the forecaster's incentives. If the prediction is for good weather and it turns out bad, people will be angry. If it's for bad weather and it turns out good, people will be pleasantly surprised. It would be very surprising if forecasters didn't err heavily on the side of predicting bad weather.

I have no data--just a theory--but if we'd had all the snow predicted for us this winter, we'd still be digging out.

Dan Krymkowski

How much money do we spend on weather forecasts each year? Is this money well spent, given the low degree of accuracy?

Vincent

A week before my wife and I were to get married, the weather forecast called for rain on our wedding day.

That prediction continued right up to the Friday before the big day. The photographer came up with backup plans to use his studio for group pictures.

Our wedding day ended up being sunny and warm. Not a drop of rain.

Douglas

Years back, in the mid atlantic states, a student of mine did a similar study of the newspaper predictions for precipitation, the day before, and found them something liken 70 percent accurate. He also found that the weather service people used a Breyer formula to judge their accuracy. I wonder if that formula is still in use.

It seems a misnomer to term the weather forecast report a piece of microeconomics; it is just using numbers to answer a question, not a matter of people responding to incentives. It is numeracy, applied.