Predictions are a dime a dozen.
Who is going to win the next election? The next Super Bowl? The World Cup? What will tomorrow’s high temperature be? What will GDP growth be next year?
The possibilities are endless. And there are endless prognostications.
Seasonal hurricane forecasting is a thing. Every year, dozens of groups around the world issue forecasts for the coming year’s Atlantic hurricane activity. In 2022, these groups — as a collective — were wrong, and comprehensively so.
The Barcelona Supercomputing Center has collected seasonal hurricane forecasts from 29 organizations which allows a detailed analysis of just how wrong the community was in 2022.
Of the 19 organizations that issued forecasts last spring, 17 of them predicted an above-average frequency of “named storms” (i.e., those that the U.S. National Hurricane Center names based on estimates of intensity). [emphasis, links added]
By early summer, 19 of these forecasts called for an above-average season.
The actual hurricane season was below average by most metrics, as you can see in the table below, via the one and only Phil Klotzbach at Colorado State University.
Seasonal hurricane forecasting of hurricanes was developed by the late William Gray (Klotzbach’s mentor and the first scientist to invite me to speak at a university, back in 1993) and has developed into a cottage industry.
But are they any good?
Forecasts can be evaluated based on a metric called “skill”, which refers to how a forecast compares to the use of a naïve baseline expectation. If your methodology cannot improve upon a naïve baseline then it has no skill and its use should be questioned.
Seasonal hurricane forecasting is challenging because naïve baseline forecasts tend to do quite well.
Consider that a persistent forecast of hurricane activity — assuming that the next year will follow the previous year in terms of being more or less active than the median — would have correctly anticipated more than 2/3 of hurricane activity in years since 1990.
Hurricane activity is streaky. The past 6 years have all been above median activity, but 2022 breaks that streak.
Forecasting above median activity would have been a winner each of the past 6 years but failed in 2022. And 1970 to 1994 (except two years) were all below the median.
Some forecasters routinely miss the mark, For instance, Penn State (Michael Mann) issues forecasts of “named storms” but the reality has fallen outside their wide forecast range in each of the past 4 years.
In general, I do not like “named storms” as a metric because it involves judgment calls by NOAA hurricane experts (on whether to “name” a storm) and is thus less rigorous a metric than satellite-derived estimates of hurricanes, major hurricanes, or accumulated cyclone energy (ACE).
Predictions of ACE for 2022 made in August — after the hurricane season had started but before the bulk of activity — were also off target.
Of the 10 groups forecasting seasonal ACE in August, all predicted that activity would be above the median of activity (from 1990 to 2021). In fact, the 2022 ACE came in at about the 40th percentile of years since 1990.
Klotzbach at CSU does a very nice job looking back at their seasonal forecasts and assessing how well they performed.
The figure below shows a fairly strong correlation between their forecasts and observations, but as I’ve noted, they have a very high bar to show skill, based on the strength of persistence and other naïve techniques.
In a 2018 paper led by Klotzbach — updating a discovery we first made in 1999 — we showed that landfalling major hurricanes occurred twice as often in La Niña years as compared to El Niño years, and 4 times as often for Florida.
With Ian’s landfall in Florida as a Category 4 storm in 2022, this relationship gets even stronger. Climate change is important, of course, but its projected effects on hurricanes are measured in a few percentage points over a century — ENSO has effects that are orders of magnitude larger in just years. Don’t lose sight of climate variability.
So what lessons should we take from the seasonal forecasting bust of 2022?
- Prediction is hard, especially about the future
- That said, there are some indications of skill in seasonal hurricane forecasts, based on simple metrics and relationships
- You can safely ignore all forecasting efforts that do not rigorously evaluate their past efforts
- CSU and Phil Klotzbach remain the head of the class
- If you care about landfalls and damage, then focus directly on the correlates of landfalls and damage
Read rest at The Honest Broker
Comments 3