Good Models and Bad Models

By: Michael Ashton | Mon, Mar 13, 2017
Print Email

I have recently begun to spend a fair amount of time explaining the difference between a "good model" and a "bad model;" it seemed to me that this was a reasonable topic to put on the blog.

The difference between a good model and a bad model isn't as obvious as it seems. Many people think that a "good model" is one that makes correct predictions, and a "bad model" is one that makes bad predictions. But that is not the case, and understanding why it isn't the case is important for economists and econometricians. Frankly, I suspect that many economists can't articulate the difference between a good model and a bad model...and that's why we have so many bad models floating around.

The definition is simple. A good model is one which makes good predictions if high-quality inputs are given to the model; a bad model is one in which even the correct inputs doesn't result in good predictions. At the limit, a model that produces predictions that are insensitive to the quality of the inputs - that is, whose predictions are just as accurate no matter what the inputs are - is pure superstition.

For example, a model of the weather that depends on casting chicken bones and rat entrails is a pretty bad model since the arrangement of such articles is not likely to bear upon the likelihood of rain. On the other hand, a model used to forecast the price of oil in five years as a function of the supply and demand of oil in five years is probably an excellent model, even though it isn't likely to be accurate because those are difficult inputs to know. One feature of a good model, then, is that the forecaster's attention should shift to the forecasting of the inputs.

This distinction is relevant to the current state of practical economics because of the enormous difference in the quality of "Keynesian" models (such as the expectations-augmented Phillips curve approach) and of monetarist models. The simplest such monetarist model is shown below. It relates the GDP-adjusted quantity of money to the level of prices.

M2/GDP and GDP Deflator 1913-2013

This chart does not incorporate changes in money velocity (which show up as deviations between the two lines), and yet you can see the quality of the model: if you had known in 1948 the size of the economy in 2008, and the quantity of M2 money there would be in 2008, then you would have had a very accurate prediction of the cumulative rate of inflation over that 60-year period. We can improve further on this model by noting that velocity is not random, but rather is causally related to interest rates. And so we can state the following: if we had known in 2007 that the Fed was going to vastly expand its balance sheet, causing money supply to grow at nearly a 10% rate y/y in mid-2009, but at the same time 5-year interest rates would be forced from 5% to 1.2% in late 2010, then we would have forecast inflation to decline sharply over that period. The chart below shows a forecast of the GDP deflator, based on a simple model of money velocity that was calibrated on 1977-1997 (so that this is all out-of-sample).

GDP Deflator, 4q Average 2006-2016

That's a good model. Now, even solid monetarists didn't forecast that inflation would fall as far as it did - but that's not a failure of the model but a failure of imagination. In 2007, no one suspected that 5-year interest rates would be scraping 1% before long!

Contrariwise, the E-A-Phillips Curve model has a truly disastrous forecasting history. I wrote an article in 2012 in which I highlighted Goldman Sachs' massive miss from such a model, and their attempts to resuscitate it. In that article, I quoted these ivory tower economists as saying:

"Economic principles suggest that core inflation is driven by two main factors. First, actual inflation depends on inflation expectations, which might have both a forward-looking and a backward-looking component. Second, inflation depends on the extent of slack (or spare capacity) in the economy. This is most intuitive in the labor market: high unemployment means that many workers are looking for jobs, which in turn tends to weigh on wages and prices. This relationship between inflation, expectations of inflation and slack is called the "Phillips curve."

You may recognize these two "main factors" as being the two that were thoroughly debunked by the five economists earlier this month, but the article I wrote is worth re-reading because it describes how the economists re-calibrated. Note that the economists were not changing the model inputs, or saying that the forecasted inputs were wrong. The problem was that even with the right inputs, they got the wrong output...and that meant in their minds that the model should be recalibrated.

Core CPI Inflation, Actual versus Predicted

But that's the wrong conclusion. It isn't that a good model gave bad projections; in this case the model is a bad model. Even having the actual data - knowing that the economy had massive slack and there had been sharp declines in inflation expectations - the model completely missed the upturn in inflation that actually happened because that outcome was inconsistent with the model.

It is probably unfair of me to continue to beat on this topic, because the question has been settled. However, I suspect that many economists will continue to resist the conclusion, and will continue to rely on bad, and indeed discredited, models. And that takes the "bad model" issue one step deeper. If the production of bad predictions even given good inputs means the model is bad, then perhaps relying on bad models when better ones are available means the economist is bad?

 


P.S. Don't forget to buy my book! What's Wrong with Money: The Biggest Bubble of All. Thanks!

You can follow me @inflation_guy!

Enduring Investments is a registered investment adviser that specializes in solving inflation-related problems. Fill out the contact form at http://www.EnduringInvestments.com/contact and we will send you our latest Quarterly Inflation Outlook. And if you make sure to put your physical mailing address in the "comment" section of the contact form, we will also send you a copy of Michael Ashton's book "Maestro, My Ass!"

 


 

Michael Ashton

Author: Michael Ashton

Michael Ashton, CFA
E-Piphany

Michael Ashton

Michael Ashton is Managing Principal at Enduring Investments LLC, a specialty consulting and investment management boutique that offers focused inflation-market expertise. He may be contacted through that site. He is on Twitter at @inflation_guy

Prior to founding Enduring Investments, Mr. Ashton worked as a trader, strategist, and salesman during a 20-year Wall Street career that included tours of duty at Deutsche Bank, Bankers Trust, Barclays Capital, and J.P. Morgan.

Since 2003 he has played an integral role in developing the U.S. inflation derivatives markets and is widely viewed as a premier subject matter expert on inflation products and inflation trading. While at Barclays, he traded the first interbank U.S. CPI swaps. He was primarily responsible for the creation of the CPI Futures contract that the Chicago Mercantile Exchange listed in February 2004 and was the lead market maker for that contract. Mr. Ashton has written extensively about the use of inflation-indexed products for hedging real exposures, including papers and book chapters on "Inflation and Commodities," "The Real-Feel Inflation Rate," "Hedging Post-Retirement Medical Liabilities," and "Liability-Driven Investment For Individuals." He frequently speaks in front of professional and retail audiences, both large and small. He runs the Inflation-Indexed Investing Association.

For many years, Mr. Ashton has written frequent market commentary, sometimes for client distribution and more recently for wider public dissemination. Mr. Ashton received a Bachelor of Arts degree in Economics from Trinity University in 1990 and was awarded his CFA charter in 2001.

Copyright © 2010-2017 Michael Ashton

All Images, XHTML Renderings, and Source Code Copyright © Safehaven.com