How does the chief risk officer of any organization know when a risk factor is significant? For most corporate risk and credit applications, the measures which are employed to illuminate an extremely rare event such as criminal acts or financial fraud are much less interesting than the cultural logic behind the internal control systems of any firm, what lawyers call Bayesian logic trees. Too often, complexity is substituted for basic common sense. Sometimes knuckle dragging simplicity is a good thing.

We like the tried and true "hands full" rule. Basically, something is a concern when it takes more than one hand to hold and becomes really important to consider when it takes more than two hands. For you numbers fans remember that homo-sapiens have five fingers a hand. That means something must stand out by at least 5% above the noise to merit notice and 10% above to raise Mr. Spock's eyebrow.

Readers of The IRA are familiar with our caution about relying on efficient market theory pricing (a.k.a. Merton models) as determinants of risk. We also wonder about the practical utility of extreme risk events (a.k.a. VAR) analysis methods which deliver dire messages - but offer managers little resolution. The numerical indicators of most VAR models indicate absolute risk shifts of � to 1 percent over baseline values; values which start at less than 1 percent occurrence probability in the first instance.

There are several flavors of the anomaly survey approach currently in use by the major financial houses working toward advanced IRB status under Basel II, but all seem to share the fatal characteristic of on-the-margin analysis. By the time the final results are tabulated, these models also often produce ridiculously high false positive rates, sometimes to the point of overwhelming the desired indicator itself by 10x-20x.

The already narrow indicators are then expanded to a one hundred percent scale and sliced and diced into quartiles, quintiles, and deciles depending on the preference of the analyst. It's important to remember we are still talking about an absolute shift of � percent. That's a 200X amplifier of a signal well into the noise floor. That's not even a nail chip on your pinky finger on the knuckle draggers scale of statistical significance.

We are cautious about "contemporary" risk measurement tools because the design feels flawed by the fact that the sequence of computations literally "blows things out of proportion." Analysts who spend time on extreme risk scenarios are probably not getting any better indication by using these hypersensitive techniques than one would gain, for example, by simply reading the newspaper -- or noting that the KMV EDF score of the subject being analyzed is now beyond 20% -- or better, just dropped off the coverage list. Risk is not the same thing as uncertainty, after all. No amount of Monte Carlo modeling will bring clarity to anticipating rare events.

Contrary to the "best guess" approach Wall Street applies to threat analysis, risk analysts at Basel banks should be going the other way - that is, looking for indicators that have over 10% shifts versus the baseline, then combining and dampening these factors to eliminate false positives. Anything that still makes a squeak will be well past the threshold required to wake up the CRO in the middle of the night. For this reason, we believe that members of academia studying methods of detecting aberrant behavior needs to refocus attention in this direction -- as Dr. Altman did in the 1960's when he invented the Z-Score.

Comments? Questions? Contact us at [email protected]