I have recently begun to spend a fair amount of time explaining the difference between a "good model" and a "bad model;" it seemed to me that this was a reasonable topic to put on the blog.
The difference between a good model and a bad model isn't as obvious as it seems. Many people think that a "good model" is one that makes correct predictions, and a "bad model" is one that makes bad predictions. But that is not the case, and understanding why it isn't the case is important for economists and econometricians. Frankly, I suspect that many economists can't articulate the difference between a good model and a bad model...and that's why we have so many bad models floating around.
The definition is simple. A good model is one which makes good predictions if high-quality inputs are given to the model; a bad model is one in which even the correct inputs doesn't result in good predictions. At the limit, a model that produces predictions that are insensitive to the quality of the inputs - that is, whose predictions are just as accurate no matter what the inputs are - is pure superstition.
For example, a model of the weather that depends on casting chicken bones and rat entrails is a pretty bad model since the arrangement of such articles is not likely to bear upon the likelihood of rain. On the other hand, a model used to forecast the price of oil in five years as a function of the supply and demand of oil in five years is probably an excellent model, even though it isn't likely to be accurate because those are difficult inputs to know. One feature of a good model, then, is that the forecaster's attention should shift to the forecasting of the inputs.
This distinction is relevant to the current state of practical economics because of the enormous difference in the quality of "Keynesian" models (such as the expectations-augmented Phillips curve approach) and of monetarist models. The simplest such monetarist model is shown below. It relates the GDP-adjusted quantity of money to the level of prices.
This chart does not incorporate changes in money velocity (which show up as deviations between the two lines), and yet you can see the quality of the model: if you had known in 1948 the size of the economy in 2008, and the quantity of M2 money there would be in 2008, then you would have had a very accurate prediction of the cumulative rate of inflation over that 60-year period. We can improve further on this model by noting that velocity is not random, but rather is causally related to interest rates. And so we can state the following: if we had known in 2007 that the Fed was going to vastly expand its balance sheet, causing money supply to grow at nearly a 10% rate y/y in mid-2009, but at the same time 5-year interest rates would be forced from 5% to 1.2% in late 2010, then we would have forecast inflation to decline sharply over that period. The chart below shows a forecast of the GDP deflator, based on a simple model of money velocity that was calibrated on 1977-1997 (so that this is all out-of-sample).
That's a good model. Now, even solid monetarists didn't forecast that inflation would fall as far as it did - but that's not a failure of the model but a failure of imagination. In 2007, no one suspected that 5-year interest rates would be scraping 1% before long!
Contrariwise, the E-A-Phillips Curve model has a truly disastrous forecasting history. I wrote an article in 2012 in which I highlighted Goldman Sachs' massive miss from such a model, and their attempts to resuscitate it. In that article, I quoted these ivory tower economists as saying:
"Economic principles suggest that core inflation is driven by two main factors. First, actual inflation depends on inflation expectations, which might have both a forward-looking and a backward-looking component. Second, inflation depends on the extent of slack (or spare capacity) in the economy. This is most intuitive in the labor market: high unemployment means that many workers are looking for jobs, which in turn tends to weigh on wages and prices. This relationship between inflation, expectations of inflation and slack is called the "Phillips curve."
You may recognize these two "main factors" as being the two that were thoroughly debunked by the five economists earlier this month, but the article I wrote is worth re-reading because it describes how the economists re-calibrated. Note that the economists were not changing the model inputs, or saying that the forecasted inputs were wrong. The problem was that even with the right inputs, they got the wrong output...and that meant in their minds that the model should be recalibrated.
But that's the wrong conclusion. It isn't that a good model gave bad projections; in this case the model is a bad model. Even having the actual data - knowing that the economy had massive slack and there had been sharp declines in inflation expectations - the model completely missed the upturn in inflation that actually happened because that outcome was inconsistent with the model.
It is probably unfair of me to continue to beat on this topic, because the question has been settled. However, I suspect that many economists will continue to resist the conclusion, and will continue to rely on bad, and indeed discredited, models. And that takes the "bad model" issue one step deeper. If the production of bad predictions even given good inputs means the model is bad, then perhaps relying on bad models when better ones are available means the economist is bad?
P.S. Don't forget to buy my book! What's Wrong with Money: The Biggest Bubble of All. Thanks!
You can follow me @inflation_guy!
Enduring Investments is a registered investment adviser that specializes in solving inflation-related problems. Fill out the contact form at http://www.EnduringInvestments.com/contact and we will send you our latest Quarterly Inflation Outlook. And if you make sure to put your physical mailing address in the "comment" section of the contact form, we will also send you a copy of Michael Ashton's book "Maestro, My Ass!"
