Bias, Variance and Loss in Avalanche Forecasting by Peter Donner
Revised: April 9, 2019
Statistical theory is relevant to avalanche forecasting through the concepts of bias, variance, and loss. Bias is the systematic difference between the forecast and actual conditions. Variance is the random difference and captures the notion that because conditions cannot be anticipated with certainty, forecast and actual can be different. Loss is the cost, however defined, of an inaccurate forecast. The best forecast is the one that minimizes loss.
PURE STATISTICAL THEORY
In pure statistical theory, which is completely abstract and ignores human well-being, loss is usually defined as a function of bias and variance. Bias, in particular, is generally undesirable because the objective of the forecast is to predict actual conditions, not to make a prediction that is expected to be different from actual. When the forecast is expected to be the same as actual, except for unknown random effects, or variance, it is unbiased. In complex problems, forecast and actual are never or rarely the same, but the process is designed so the forecast is high as often as it is low and over time the difference averages out to zero. If the forecast is unbiased, then the loss is simply the variance and the objective is to design a technique that minimizes variance.
AVALANCHES HURT PEOPLE
Avalanche forecasts are not abstract, and a forecast that understates the actual hazard can result in injury or death. In this situation, an unbiased technique, one that understates the hazard as often as it overstates, may be undesirable in the sense people can be injured or killed when the actual hazard is greater than forecast. In contrast to pure theory, then, an upward or positive bias in avalanche forecasts may be desirable.
DANGER SCALE AND ROSE
While experienced backcountry users typically look at the weather forecast before the avalanche forecast, even the most experienced focus on the “current danger rating,” the one-word summary of the overall hazard, based on the North American Public Avalanche Danger Scale. On this scale, an experienced user may say
1.Low green go—human triggered avalanche unlikely
2.Moderate yellow go carefully— human triggered avalanche possible 3.Considerable orange think about where to go— human triggered avalanche likely 4.High red light keep it low angle— human triggered avalanche very likely 5.Extreme black keep it very low angle— human triggered avalanche certain
After the one-word summary, the danger rose is highlighted in official forecasts produced by the UAC. Comprised of 8 aspects and 3 elevations, it has 24 petals, each of which gets a danger rating. When the rose is all green petals, users think of visiting terrain they would never go to when the danger is considerable.
Objective analysis of summary conditions and of snow and weather and recent avalanches adds nuance to the forecast. This analysis is difficult to quantify in terms of bias and variance, so the discussion will be limited to the numeric danger scale, 1=low, etc.
AVALANCHE MECHANICS
Avalanches result from the interaction of four elements: 1) a slab of snow is 2) triggered 3) to fail on a weak layer and 4) slides downhill on a bed surface. The problem in the human context is that people make excellent avalanche triggers. In terms of developing a forecast danger rating, uncertainty exists about slab formation and to a lessor extent weak layers and bed surfaces.
SNOWPACK UNCERTAINTY IS KNOWN
Snowpack is relatively well understood by UAC forecasters, though some uncertainty exists about the nature and location of slabs, weak layers and bed surfaces. Combining its staff and its observer network, UAC receives information from hundreds if not thousands of snowpits during the season. Beginning with first snow in fall and continuing through winter into spring, UAC monitors weak layers on the ground and throughout the pack, noting which are dormant and which are actively producing avalanches.
THE REAL UNCERTAINTY IS WEATHER
The real uncertainty in the daily forecast is weather, what precipitation intensity (PI) and wind will be over the course of the coming day. Though it is true during warm spells and as spring commences, air temperature and direct sun can turn dry snow wet, for the present discussion the focus will be on cold winter conditions where PI and wind are the main weather variables. Given enough loose snow on the ground, sustained and gusting wind on its own can increase the hazard from low to considerable in a few hours, sometimes in a few minutes. Likewise, high PI combined with any wind can spike the danger 1 or 2 levels. Because the UAC is located at the Salt Lake office of the National Weather Service, it has the best available information on likely weather as the daily avalanche forecast is published. This serves to minimize forecast variance.
LOSS FROM FORECAST ERROR IS ASYMMETRIC
If loss, the cost of an inaccurate forecast, is defined in terms of damage to backcountry users, rather than variance from actual, then it is asymmetric. Although 1 and 5 are both a distance of 2 from 3, that is, low is two steps below considerable, and considerable is two steps below extreme, the expected loss from forecasting 1=low when conditions are actually 3=considerable is greater than the loss from forecasting 5=extreme.
The reason for the asymmetry is that people are more likely to travel in hazardous terrain when the forecast is low than when it is extreme. If actual conditions are considerable but the forecast is low, then more people will travel in hazardous terrain, making accidents and harm are more likely. In contrast, if the forecast is extreme, few, perhaps no, people will travel in hazardous terrain. While no, or less, physical harm occurs when the forecast overstates danger, those with exceptional hazard management skills will have missed the opportunity for more challenging and rewarding recreation. Over time this unnecessary loss of opportunity will cause users, especially advanced users, to lose
respect for the forecast. Nonetheless, by any reasonable measure, the loss of life is more costly than the loss of respect.
Asymmetric loss, then, suggests the forecasting procedure should be designed to minimize situations where actual conditions are more hazardous than forecast. In particular, the procedure should only generate low hazard when the actual danger during the coming day is low. A low forecast causes people to visit hazardous terrain. If it is possible winds or PI higher than forecast would lead to a higher hazard, then the forecast should not be low.
A BIAS TO OVERSTATE DANGER IS GOOD
None of this discussion is unknown to UAC, but framing the issue in terms of bias, variance and loss may enable better communication of how the forecast relates to actual hazard. To some extent, UAC already incorporates positive bias into its forecasts, so when the forecast is different from actual, forecast hazard is higher than actual much more often than lower. This practice has lead users, particularly advanced users, to view the UAC as too conservative and to discount the forecast hazard. Comments like “considerable is the new moderate” are sometimes heard. And yet serious injuries occurred with all members of the touring party caught and carried in an avalanche on two occasions during 2019 when actual conditions were considerable or high and the forecast was low or moderate.
OBJECTIVES AND CONSTRAINTS IN AVALANCHE FORECASTING
The situation can be framed in the terms of classic optimization, where an objective is optimized subject to constraints. The constraints must be first satisfied and then the best possible is done with the objective. If the focus is providing accurate information, the problem is to minimize forecast related accidents subject to the constraint the forecast informs backcountry user decision making. Here the constraint is to inform decision making, and some forecast-related accidents are expected to occur. If, in contrast, the focus is preventing forecast-related accidents, the problem is to maximize information subject to the constraint no forecast-related accidents occur, or expected accidents are zero. This is a sketch of the logic with the understanding the application is ill-posed because the objective and constraints cannot be specified as a mathematical function of data UAC possesses.
It is worth considering how the constraint expected forecast-related accidents are zero could be made operational. The main point is when the forecast is accurate, it does not cause accidents. If the forecast is considerable, for example, and this verifies during the day, then if accidents occur it is because users traveled in terrain that was forecast as likely to avalanche. In this case the accident was caused by user error, not forecast error. Likewise, if the forecast overstates hazard, then accidents that occur are, by definition, the result of user error. It is only when the forecast hazard is lower than actual that the error in the forecast may cause accidents.
ENHANCED WEATHER AS A SAFETY FACTOR
If, as suggested above, the main source of forecast error is weather, then one way to eliminate expected forecast-related accidents is to incorporate cases when wind or PI were higher than forecast. I’m not suggesting UAC do this, just continuing to sketch the logic necessary to consider the problem. One approach would be to establish a window around the forecast date and analyze how often wind or PI was higher than forecast. If the window is two weeks and the forecast date is January 10, then data for January 3 to January 17 for some period of years would be analyzed. The National Weather Service uses
30 years to establish weather norms. The result is what is known in engineering as a safety factor that is added to the previous standard forecast wind and PI. This could be called the enhanced weather forecast in contrast to the standard forecast the UAC has previously used.
Enhanced weather is most useful in eliminating forecast related accidents when standard weather suggests the forecast danger will be low, and to a lessor but still significant extent with moderate danger. When avalanches are forecast to be unlikely, or danger is low, reasonable people will consider visiting hazardous terrain. When avalanches are possible, or danger is moderate, it is still reasonable to consider visiting hazardous terrain, but fewer people will and those that do will be more mindful of mitigation strategies. When standard weather leads to a forecast of considerable or higher danger, since avalanches are likely or certain, there is no need for a safety factor because reasonable people will develop strategies to mitigate the hazard.
CONCLUSION: HERE BE WOLVES
Again, UAC forecasters are well acquainted with the concept of enhanced weather as a safety factor though they may not use the phrase and there may not be a shared understanding among the staff on how to incorporate the logic into the forecast. In the two accidents in 2019 where the entire party was caught and carried, the suggestion that wind and PI higher than forecast could spike the danger 1 or 2 levels might have caused the parties to develop mitigation strategies that would have prevented the avalanches. The downside to this approach is that it is a bit like crying wolf. Both forecasters and users will grow tired of a warning that is never needed. But even the boy who always cried wolf ultimately did come across a wolf.