Those are the words of Warren Buffet, who warned of the coming credit crisis. Buffet—one of the very few—had little faith in the “complicated, computer-drive models systems that many financial giants relay on to minimize risk.”
AIG built a lot of models which attempted to quantify risk and uncertainty in their financial instruments. They, like many other firms, tried to verify how well these models did, but they only did so on the very data that was used to build the models.
Now, if you are a regular reader of this blog, you will know that we often talk about how easy it is to build a model to fit any set of data. In fact, with today’s computing power, doing so is only a matter of investing a small amount of time.
But while a model fitting the data that was used to build it is necessary condition for that model to work in reality, it is not a sufficient condition. Any model must also be tried on data that was not used—in any way—to build it.
What happened at AIG, and at other financial houses, was that events occurred which were not anticipated or that had not happened before. Meaning, in short, that the models in which so many had so much faith, did not work in reality.
There is only one true measure of a model’s value: whether or not it works. That it is theoretically sound, or that it uses pleasantly arcane and inaccessible mathematics, or that it matches our desires, or that “only PhDs can understand” it are all very nice things, but they are none of them necessary. Many complex models which are in use are loved and trusted because of these things, but they should not be. They should only be valued to the extent that they accurately quantify the uncertainty of the real-life stuff that happens (climate models anyone?).
What the AIG models failed to account for were the “unknown unknowns”—to use Donald Rumsfeld’s much maligned quotation. They did not quantify the uncertainty of events which they did not know about. They thought that the models quantified the uncertainty of every possible thing that would happen, but of course they did not. Meaning that they were overconfident.
AIG’s failure is yet another in a long series of lessons that the more complex the situation, the less certain we should be.
(A subscription is required to read the full WSJ article.)