The stakes for risk management decisions are higher than ever and thus these decisions need to be supported by a higher level of detail. Determining that appropriate level of detail is a matter of understanding and grappling with the uncertainties of probability that inevitably challenge the risk management profession. We need to be able to critically assess the applicability and credibility of all sources of risk management information—from the numbers (the base of all risk analysis) to the intuitive powers of the risk decision makers.
A Lack of Data
To make sound judgments based on statistics, we need data. The more data the better. For most organizations, however, data on individual exposures is scarce. This is especially true for severe, yet infrequent events, which involve the most difficult and costly decisions.
The results of statistical study on losses for any particular exposure boil down to a chart like that shown in Figure 1. A probability number, here expressed as a percentage, is associated with various loss potentials. Likelihood is typically shown as a probability of exceeding a given annual aggregate loss.
The numbers are derived in various ways. The most common method matches theoretical equations with observed loss data. The fewer data points there are, the more possible mathematical equations fit the data. Yet only one true equation exists. Which one is correct? The choice is based on which equation minimizes the difference between its theoretical predictions and the actual data. If the actual data is subject to random fluctuations, however, the chance that the selected equation fits the true underlying loss distribution is slim.
We can take the analysis one step further and calculate the likelihood that the probability we come up with is true—our statistical confidence intervals. Unfortunately, this can lead to an infinite regression of probabilities on probabilities, and it assumes certain theoretical conditions. In this fast-moving and complex world, we can never be sure of such assumptions.
Loss prediction precision fades at exactly that point where we need to make significant decisions. How reliable is a decision based on a number developed in a system so dependent on possibly flawed theoretical ideas? To effectively use formal risk assessments, we must understand the numbers, any uncertainties surrounding them and what we can do about those uncertainties.
Actuarial Limitations: The Ten Percent Rule
While we might specify probabilities accurately for frequent events, accuracy falls off as the frequency of loss decreases. Likewise, as the potential losses increase, we become less sure of the probability of exceeding these amounts. Estimates involve picking one of the many theoretical models that predict future loss, but that choice, beyond a certain point, becomes arbitrary.
The probability number associated with a particular loss amount could be the result of a range of modeling options. The problem is compounded when the fitted models are used to extrapolate beyond observed data. Such data deficiencies mean that the results of statistical risk assessments can be highly inaccurate for losses where predicted probability falls below ten percent.
In the study shown in Figure 1, aggregate losses for our hypothetical firm are predicted to exceed $6.6 million 10 percent of the time (or once every ten years). So losses will be under $6.6 million 90 percent of the time. Results at higher thresholds—5 percent or 1 percent—are highly questionable, regardless of the skill of the statistician. This means that if we are concerned about losses exceeding $8 million, or we want a comfort level in excess of 90 percent with our risk financing decisions, extra caution should be used.
This problem manifests itself when we are dealing with more sophisticated risk management and risk financing issues, beyond buying insurance or selecting a maintenance deductible. The hard data on which to make such decisions may not exist. We may need to be satisfied with approximations.
Real World Decisions
Astute risk managers have always recognized the imperfection of the data they deal with. The uncertainty of unknown data is remedied by using tried-and-true rules of thumb.
For example, retentions are set where the probability of exceeding a certain loss amount drops below 5 percent or 10 percent a year. This is the classic linguistic rule for retention: Retain frequent losses. With credible data at or near the 10 percent threshold, decisions can usually be made with confidence. The only potential anxiety arises from the desire to make such decisions more exact or scientific; seasoned risk managers take comfort in knowing that we just cannot do this.
Intuitive and formal methods for more advanced financing decisions, however, are underdeveloped. Estimated probabilities in such cases can be used as inputs for loss simulations, in which possible losses are generated by running a model through various sample years. Risk management strategies can be superimposed onto the simulation and the results noted over time to assess the ultimate cost, variability and safety of various strategies.
But the old adage of computer modeling stands: garbage in, garbage out. The results of our model depend critically on the veracity of the model inputs—in this case, the estimated loss probabilities. Simulation itself does not lend credibility to the model. It reflects the perceived reality; if that perception is wrong, the results are wrong.
At high loss levels, we practice the principle of precaution: If large losses are at least possible, take the necessary precautions. This means buying insurance and using loss prevention and control to eliminate the possibility of harm. We know that certain things can harm the company and others cannot, and that there is an uncertain layer between; we must identify and deal with this layer. Formal methods of predicting outcomes from the hazy layer of unknown data can actually mask uncertainty behind seemingly precise risk estimates. Such estimates may be handpicked from the arbitrary spectrum to support one or another risk management alternative. To avoid this, we need to establish a threshold for “practical impossibility,” which may not necessarily be based on hard numbers.
The intuitive abilities of those familiar with loss exposures must not be overlooked. Engineers, insurance underwriters and risk managers have a good feel for loss probabilities. Our intuition is the distillation of hard-to-articulate experience and is often our best guide in assessing the credibility of risk studies. Combining the results of formal studies with intuitive judgments maximizes the accuracy of each method.
Making a Stronger Case for Risk Analysis
A variety of risk assessment methods, in addition to intuition, can supplement statistical assessments of risk and support the much needed details of credibility to back a risk management decision.
Scenarios. Scenario-based methods construct a logical progression of loss event possibilities. For example, a fire exposure could be traced from its initiation (e.g., careless smoking) to possible outcomes. Along the way, probabilities of subevents (e.g., the functioning of sprinklers) are assessed. Ultimate fire damage outcomes are thus developed.
The scenario-based approach avoids some of the problems of statistics because the probabilities of subevents can be calculated more accurately (e.g., there is considerable data for failure rates of sprinklers). Scenario-based assessments can push the credibility of our probability estimates considerably beyond the 10 percent threshold.
Industrywide Data. Using industrywide data in risk studies lends to their credibility. Greater statistical accuracy, however, comes at the cost of relevance. As the range of the study grows, the similarity of generic data to that of the specific organization decreases. Few formal methods exist for enhancing an individual organization’s data to fit generic data. Those that do, such as Bayesian statistics, make strong theoretical assumptions that may or may not hold true.
Consultants. In considering critical exposures and related risk management decisions, we may also seek outside help in both the methods of risk assessment and their application. We cannot, however, abrogate our responsibility for making adequate risk decisions on the part of the organization. That means knowing the weaknesses and strengths of outside consultants and their methods. The true value of an outside assessment of risk is in the genuine information it provides. This information should be considered an additional information source, to be combined with intuitive judgment. In judging the true worth of formal assessments, we must realize that such assessments do not supplant expert judgment, or vice versa.
Treating Uncertainty. Formal risk assessments can be greatly improved if they explicitly incorporate uncertainty due to knowledge imperfections. This cannot be accomplished using the standard methods of probability and statistics. Confidence intervals assess uncertainty due to randomness in the sampling process; they do not address the idea of knowledge imperfection directly.
The simplest way to suggest uncertainty due to knowledge imperfection is the use of intervals of uncertainty, suggested by the multiple plausible alternatives shown in Figure 2. The credibility of these intervals is judged by how well they let us deal with the real world.
The balance is between specificity and truth. If data is scarce, and we create an exact estimate, it is less likely to be true.
No Easy Answers
Modern risk management is not just about assessing probabilities and making the appropriate cost/benefit decisions. This greatly oversimplifies the process. A huge part of risk management is understanding and dealing with the unknown. Formal risk assessment methods help us realize what we do not know and what we do. The risk of being misled by false precision, however, remains very real.
