Salil Mehta at Statistical Ideas commented on my recent post that compared the Atlanta Fed GDPNow model to the FRBNY Nowcast model.

Mehta notes that both models claim accuracy within one percentage point. However, that's no longer mathematically possible given the difference between the two models is 2.3 percentage points.

Rogue one: Faithful GDP nowcasts by Salil Mehta

There is a 2/3 chance that both competing Federal Reserve 2017 Q1 GDP nowcasts are wrong! That's an audacious prediction for the storied NY and Atlanta institutions (one of them led by my former big boss Timothy Geithner), and yet there is no way around the current confusion they are in. This is also critically important as one is showing a robust 3.2% growth reading, while the other is at 0.9% (the 2nd lowest reading in nearly 3-years) and essentially indicates that we are descending towards recession.

Are we descending towards recession? While we don't forecast that, we certainly think there is only a single digit probability of a >3% GDP. How could the NY Fed plausibly give such a madly high estimate (which if true would be the second highest in 2-years)? Yet, there you have it, two extreme readings, and a 2.3% (3.2%-0.9%) chasm between them.

We show here that the Federal Reserve's conclusions are somewhat ridiculous, though shouldn't be since they impact the open market committee monetary decisions that the world looks to. And there are humbling lessons from these nascent Big Data, overfit models.

The chart here shows some basic information regarding the current GDP nowcasts. As we via the two blue bars, we have the Atlanta nowcast on the left (the bar was recently as high as 3.4% earlier this year). And the NY nowcast on the right (the bar was recently as low as 1.5%). That's right, both nowcasts passed each other, while aggressively moving further in the opposite direction! The large swings in each are also doubtful, given each nowcast's eventually advertised, margin of error. For a good chronology of these nowcast reports, refer to MishTalk.

Each nowcast boasts a margin of error of just ~1%, and this clearly poses a problem since the average of these two nowcasts (shown in orange at 2.1%) is clearly outside of both the Atlanta and the NY stated margin of error! As supportive reference, we also show (in green) that the current 2016 Q4 GDP is nearby at 1.9%. Now we should ask some important questions about how we keep getting into more strange nowcasts in the past year that they both have operated. The first thing to appreciate is that the nowcasts are supposed to predict very tight errors that are uncorrelated to the variance in the actual GDP itself. And good nowcasts should have errors independent of one another, except since the NY and Atlanta Fed operate independent of one another there is a good chance that there may be some modeling similarities. We modestly assume this and derive through the variance formula (VarianceAtlanta+VarianceNY+2?Atlanta?NY?Atlanta,NY) that the margin of error of the difference between the models is just less than 1% (silver vertical interval arrows in chart above). This is a highly plausible tight expected variance. Sample size is also trivial here as we don't have the true expectation to model a limit from. And with this, the probability of seeing an inadvertent 2.3% difference between the two correct Federal Reserve models is <5%. Or that their publicized margin of error is awkwardly too low (to the point we'll show that randomly guessing the GDP would be safer).

We also have the probability that one of the nowcasts is correct, which due to symmetry means applying one of the nowcast stated margin of errors to the other nowcast value. Or that the one nowcast is unintentionally correct, which would be like the <5% probability above. So, in total the probability of one of the Federal Reserve nowcasts being correct and the other being wrong is ~10% (not the >½ that many may generously assume as they critique these divergent outputs).

That leaves us with two other possible outcomes still! That is to split up the remaining 85% probability that both models are individually wrong into: (a) the average of the two models is still correct, and (b) even the average of both models is wrong. This leaves us with no choice but to conclude that the probability that there is ~40% probability that the correct 2016 Q1 GDP is nowhere near either nowcasts nor the average of the two, and a >½ probability that the GDP is near the 2.1% average that inappropriately happens to be well outside both two nowcasts' margin of error. And between those two we can safely claim that there is a 2/3 chance that both models are total wrong (and merely <5% chance they are still both right). There is perhaps a 30% chance one can smartly use both models in a deliberate way, though this is conditional on how they use the information and not at their endorsed specified face value.

Now a more practical assumption of this is that the margins of error should be more than doubled (to 2.6%!), in which case no one would even use such nowcast models. However, the probability breakdown in such a scenario is this:

• <30% chance both models -with their current 2.3% chasm- are correct
• ~60% chance one of the models is wrong and one is correct
• ~10% chance both models are still wrong individually, though the average in rare cases is correct

In all cases, all three probabilities sum to 100% as they should. And they lend a healthy sense of respect that incessantly observing each of these GDP nowcasts is commonly a waste of time, and that rarely will one gain insight from it other than from ex post luck. It's the same as the more senior open market committee models vainly attempt to forecast other macro-economic variables. Sometimes simply looking at the most recent quarter's GDP (in this case ~2%) is as good of a guess as any. So is giving a little more weight to the near-0 probability most have of an outright contraction this quarter. As a business CEO, one would want to be arranged for anything at this point, which a genuine 2.6% margin of error about GDP infers.

End Salil Mehta

Thanks Salil

I have been following this divergence for some time. I expect model revisions following the next GDP report.

I last covered the divergence in GDPNow Forecast Dips to 0.9%: Divergence with Nowcast Hits 2.3 Percentage Points – Why?

Here are my charts.

Comparison Details

Neither report had a significant movement on February 24. Let's start there for a closer look.

1. On March 1, Construction spending took 0.7 percentage points off GDPNow but only 0.046 percentage points off Nowcast.
2. On March 2, light vehicle sales took 0.3 percentage points off GDPNow. Nowcast did not factor in light vehicles sales.
3. On March 6, the manufacturing report took 0.2 percentage points off GDPNow. The same inventory report (different name) added 0.031 percentage points to Nowcast.
4. On March 7, the import-export trade report did nothing for GDPNow. The import-export trade report added a net 0.032 to Nowcast.
5. On March 10, the jobs report subtracted 0.3 percentage points from GDPNow. The Jobs report added 0.003 (nothing) to Nowcast.

Comparison Summary

• March 1, 2, 6, and 7 reports subtracted 1.2 percentage points from GDPNow.
• March 1, 2, 6, and 7 reports added 0.017 percentage points to Nowcast.
• The big difference is in how the models treat (or don't treat at all), construction spending and light vehicle sales.
• The jobs report decline (an additional 0.3 percentage point decline for GDPNow) may be related to a model change.

I have a simple rule: Whenever one of the models seems ridiculously high, I place a mental bet against that outcome.

Last quarter I leaned towards the Nowcast forecast. This quarter I think GDPNow will be much closer.

Many recent economic reports have been weak. And failure to take into consideration vehicle sales when they have been holding up retail sales seems like a mistake.

However, construction spending, which took 0.7 percentage points off GDP Now, is so volatile and so frequently revised that it is hard to have much faith in that particular slide.

Finally, one additional problem with both models is they do not think. For a synopsis, please see Formulas Don't Think: Investigating Weather-Related GDP.