With gold forging glorious new record highs (in nominal terms), traders' interest in this metal's 8-year-old secular bull is ballooning rapidly. And like most commodities, the lion's share of gold trading happens in the futures markets. So the tactical gold-price action at any given moment is usually dominated by futures buying and selling.

Knowing this, traders often scour futures reports to look for insights into gold's current price trend. Is it likely to be sustainable long enough to profitably trade, or is a reversal imminent? While there is endless futures data available for analysis, the granddaddy of all futures reports is certainly the Commodity Futures Trading Commission's Commitments of Traders Report.

The CFTC's CoT, released each Friday with data current to the preceding Tuesday, is a devilishly-complex document. The aggregate futures CoT for 2008 alone has 126 columns of data, 5631 rows, and weighs in near 5mb in Excel format! Analyzing this beast is certainly not easy, which is probably why very few traders undertake this themselves. They generally wait for professional analysts to decipher and interpret it.

The CoT's inherent complexity has cloaked it in something of a mystic awe. Since it is so time-consuming to understand, a kind of priesthood has sprung up to translate it for the masses. As with any small group holding seemingly impenetrable esoteric knowledge, entirely too much credit is given to this priesthood. Unfortunately most traders simply accept professional analysts' CoT interpretations without question.

This is certainly true in the gold market. Having actively traded gold since this bull began in early 2001, I can't even count the number of times I've read comments like the following. "Gold open interest fell by 10,000 contracts, the smart money must be getting out." "Commercials' record short position is capping gold, it is going to fall." Note that each statement starts with fact, but concludes with mere opinion.

After having studied the weekly gold CoT numbers on and off for many years now, personally I don't find this information useful at all for my own profitable trading. At best, the CoT reports are a minor secondary gold-price driver and even then very infrequently. But since the CoT priesthood's interpretation of this complicated report can definitely sway prevailing psychology from time to time, traders need to understand it.

According to the CFTC, the precursor to the CoT was born in 1924 when the Department of Agriculture started publishing annual reports on hedging and speculation in grain futures. In mid-1962, the modern CoT era began when monthly reports were issued for 13 agricultural commodities. More futures were gradually added over time. The CoT started to be published twice a month in 1990, and weekly in 2000.

Prior to 1995, the CoT was only available by paid subscription. I suspect this really added to its mystique. Newsletter writers would buy this complex difficult-to-understand report, try to interpret it, and then sell the results in their newsletters. A letter writer really appeared to be in-the-know if he could wade through the deep waters of the CoT and formulate trading strategies based on it. But today it is freely available to all on the CFTC's website.

While there are more esoteric metrics in the CoT like concentration ratios (how few traders are holding futures), most analysis today concentrates on a couple key metrics. The first is open interest, or how many futures contracts are outstanding. The second is how these open contracts are divided among three classes of traders. Many myths have sprung up around both metrics that mislead traders.

This first chart details the open interest in COMEX gold futures since 2001, the year this secular gold bull stealthily started steaming higher. Open interest is measured in contracts, and each contract represents 100 troy ounces of gold. Since CoT data is only available once per week, all this gold price data is weekly too. In order to understand gold futures open interest, this number's historical context is utterly crucial.

The blue line is the weekly gold price, and the red line is the weekly gold futures open interest directly from the CoT. Digesting this chart will give you context to help evaluate the priesthood's interpretations of developments in OI. For example, back in mid-2001 when there were only 100k contracts of gold trading, a 10k swing in one week was a big deal (10%). Today with 450k contracts, this same 10k swing is pretty trivial (2%).

Note that OI has trended higher on balance over this entire gold bull. This is the first key lesson of CoT analysis. As any bull matures, as any price marches ever higher, it attracts more traders and more capital. This is a core tenet of the financial markets in general. Nothing begets more interest in an asset like relentlessly rising prices. This dynamic forms the virtuous circle that drives all secular bulls higher.

Back in May 2001 when I first recommended buying physical gold coins as a long-term investment in my monthly newsletter, I was ridiculed. Gold was in the mid-$260s and had languished for decades. Even in the contrarian circles in which I traffic, few wanted to touch it. But over the years as gold's price meandered higher, more traders grew interested as expected.

They bought, driving the price even higher, which caught the attention of still-wider populations of traders. Then they too bought, strengthening the virtuous circle. And lo and behold today gold is generally accepted over $1000 when it was universally loathed at a far-superior entry price under $300 some 8 years ago. Bull markets are magnetic, they attract in increasing numbers of traders and capital over time.

So for gold futures OI, we should expect this number to continue to grow on balance until this secular gold bull gives up its ghost. That won't happen until gold's global supply-and-demand fundamentals turn bearish, virtually impossible for many years yet. Gold's OI grew in a tight secular uptrend for years, until the commodities surge starting in late 2007 and stock panic in late 2008 temporarily blew it out of trend.

This shatters a common OI myth from the CoT priesthood, that record-high gold OI is always bearish for gold. In its mild form, analysts claim record gold OI levels warn of an imminent pullback or correction. This is certainly true at times. But in its extreme form, I've heard analysts claim some particular OI record means this secular gold bull is coming to an end. That is just nonsense. Check out the OI records above compared with gold itself.

Sometimes, record-high OI does precede a correction. Early 2003 and early 2008 are great examples of this. But other times, despite record OI gold soars higher. In mid-2007, gold OI first approached 425k contracts. I remember well CoT analysts at that time claiming gold had to correct hard because OI looked frothy. Yet from those $675 levels, gold soon soared to $1000 in early 2008 (and gold OI went even higher).

Usually OI does rise when gold is strong, which makes sense. Rising prices attract in more futures traders looking to ride the rally. But this is certainly not always the case. One of gold's strongest uplegs of this bull ran from late 2005 to mid-2006, from roughly $425 to $700. Yet during what was the best run in the entire bull to that point, gold OI stayed flat.

While the principle of higher gold prices leading to greater gold futures open interest is true, at a tactical level this really isn't particularly useful for trading. Why? First, gold OI usually runs concurrent with the gold price. So if OI is contracting, odds are gold is already falling. The price itself already gave traders all the warning they need. Second, there are plenty of exceptions in this bull where gold OI did its own thing and ignored gold.

So next time an analyst makes a statement like "gold open interest soared (or plunged) 10,000 contracts", keep a few things in mind. The actual number of contracts is irrelevant since OI is growing on balance. Convert the raw number into a percentage, which is meaningful. The bigger the percentage gain or loss, the greater the potential relevancy of the swing. But no matter how large the swing, at best OI moves with gold and hence offers no lead time to traders. And there are plenty of exceptions to this concurrent rule, when OI decouples from the gold price.

Although the priesthood's often-poor interpretations of OI swings mislead many traders, the confusion created by the classifications of traders is at least an order of magnitude greater. There is a small but vocal minority of analysts who are unbelievably paranoid about who is holding which futures in the gold market. Sometimes I wonder if this aspect of the gold CoT is a bigger bogeyman than central-bank gold sales.

The CoT breaks futures traders into two major categories, with any excess spilling over into a third smaller plug category. On the actual reports, these are known as non-commercial traders, commercial traders, and nonreportable positions. These are easiest to understand starting at the commercials, the CoT category that constantly frightens the less-well-informed traders in the gold world.

Commercials are large traders that are theoretically actually producing or consuming real physical gold. I say "theoretically" because this line is sometimes blurred through intermediaries. Commercials, since they traffic in real gold, often choose to lock in prices. A gold miner, in order to secure a loan to build a new mine, may be required by its bank to sell gold futures in order to lock in and guarantee the price it will get for some of its future gold. A futures contract simply sets a price today for a transaction in the future.

This hedging happens on the buy side too. A jewelry manufacturer will buy gold futures in order to lock in the price it will pay for the raw material it needs in order to fashion its finished goods. So don't fall into the trap of assuming that all commercial hedgers are always short, locking in the price for future gold sales today. Many other commercial traders are long, locking in the price for future gold purchases today.

The second major CoT category is the non-commercial traders. These are the speculators, they are buying and selling gold futures in the simple pursuit of trading profits. They have no intention of ever delivering or taking delivery of physical gold. Like all speculators though, they provide a valuable service to the hedgers. They keep the gold market liquid, effectively buying the risk that the hedgers seek to sell.

Once the commercial hedgers and large speculators are categorized, there is a small group left that fits neither classification. These are small traders with positions so little they don't have to report them to the CFTC. While the odd small-business hedger slips in here (like a farmer selling a relatively small amount of wheat futures as a hedge), this non-reportable group is dominated by small speculators.

The weekly Commitments of Traders Report documents the ongoing interplay between these three groups. Unfortunately, the relationship of commercial hedgers, large speculators, and small speculators to the gold price is widely misinterpreted and generally pretty misunderstood. I suspect this is because the CoT's numbers are often discussed without considering how futures markets actually work.

Futures trading is a strict zero-sum game. Every dollar one futures trader earns is a dollar lost by another futures trader, a direct transfer. The reason futures exchanges exist is to strictly enforce trading to ensure each trader has the means to bear any losses incurred. And as a zero-sum game, every futures contract is between two traders. One trader takes the long side while the other takes the short side of the same contract. Thus total longs and shorts are always perfectly equal!

While longs and shorts always balance out over gold futures as a whole, across the three categories of traders net positions vary. For instance, in the latest CoT report, large speculators (non-commercial traders) had 253k long contracts and 22k short contracts. If shorts are subtracted from longs, the large speculators are "net long" 231k contracts. And if this group is net long, another has to be net short. And this is virtually always the commercial hedgers.

This next chart looks at the net positions in gold futures held by the three groups of traders over this secular gold bull. Large speculators are rendered in orange, small speculators in red, and commercial hedgers in yellow. And as a zero-sum game, with longs and shorts always being equal over the entire gold-futures realm, if you add all three groups' net positions together for any given week you will always get zero.

The group the CoT priesthood is always using to mislead traders is the commercial hedgers, so we'll start there. Almost constantly a small minority of analysts is lamenting some development in the commercial-trader space leading them to believe that gold is either going to be capped or fall sharply. The most common theme I've heard over the years is "record commercial short position bearish for gold".

First of all talking about "commercial shorts" is misleading alone, since it is commercial net shorts that are relevant. Commercial longs always exist too. Anyway, this myth that record commercial net-short positions are harbingers for an imminent collapse in gold are particularly relevant right now. As you can see, a couple weeks ago the commercial net-short position soared to a record 288k contracts! In early September, this number rocketed 25% higher in a single week alone!

For the CoT priesthood, a 25% surge in commercial net shorts in one week, to a new record high no less, is the most bearish scenario imaginable. The noise about how the commercials were "going to cap gold" was deafening, and with gold already up at $994 it felt too high to many. Yet this week, just a month later, gold had surged to $1042 by Tuesday (the day the CoT is finalized).

Obviously a record commercial net-short position didn't presage a sharp tumble in the gold price. And this ties in with why CoT analysis is often so misleading. Analysts will comment on this stuff, a 25% rise to a record in one week, but offer no historical context. If you study the chart above, you can find plenty of times when new record commercial net shorts did not precede a big correction in gold.

In late 2005, commercial net shorts surged to their 5th major record of this bull. Yet right then gold had just launched a powerful rally that would take it from around $475 to $700. Similarly in late 2007 commercial net shorts surged to another record, pushing what was then an unbelievable 250k contracts. Yet gold still managed to rally strongly from around $825 to $1000 right out of that commercial net-short record.

While there were times when gold did indeed carve a short-term interim high when commercial net shorts reached a bull record, there were just as many times when gold powered on higher despite such a record. Given this statistic's bipolar behavior relative to the gold price over this bull, it is simply not a high-probability harbinger of any move. Out of the 8 records noted above, only 4 coincided with major interim gold highs. With just 50/50 odds for success, it amazes me that anyone still cares about this CoT metric.

Remember that futures open interest grows as a bull matures, which is natural as more capital seeks to deploy into the winning asset. And since every individual futures contract has a long and short side, by necessity the net-short position of at least one of the three trader groups has to grow along with open interest. Therefore, as this secular gold bull continues higher, new commercial net-short records will continue to be set.

Since every commercial net short has an offsetting net long in one of the speculator groups, why not look at these records from the other side of the same coin? Claiming record net shorts is the same thing as claiming record net longs! When speculators get excited enough about gold to drive their net longs to new records, perhaps gold is temporarily overbought. Pullbacks and corrections are healthy and normal within even the most powerful bulls, they keep sentiment balanced. Periodically bleeding off excess greed really extends the ultimate duration of bull markets.

Another problem with reading too much into the perpetual interplay between these three groups is the lines between them are blurred and arbitrary. For example, imagine a Wall Street bank has clients it trades for that include gold miners (commercial hedgers) and pure speculators. And this bank trades in its street name, as is common. Does the CFTC classify this bank as a commercial hedger or large speculator? And of course this classification can change over time for individual traders or aggregators like banks.

I suspect the CFTC's delineation of these groups is woefully inaccurate and constantly in flux. And if the groups aren't accurate, or even consistent over time, then looking at their net-long and net-short positions is even less relevant for traders. Once again, the CoT data isn't anywhere near as useful for real-world trading decisions as the priesthood likes to imply to its followers.

At Zeal, we've been profitably trading this gold bull since its earliest days. In order to make good trades ourselves, and help our subscribers thrive, we work hard to study everything in context. I wouldn't dare make a statement about the CoT to our subscribers without first investigating what had happened in similar episodes in the past. We even update high-resolution versions of this essay's charts each week on our website for our subscribers so they can follow this gold CoT data in context.

In our acclaimed monthly Zeal Intelligence newsletter, my mission is to help traders like you become informed investors and speculators able to thrive in any market. We are not swayed by anything but hard research into what is actually driving the price of any asset we are interested in trading. If actual data doesn't support popular myths, then they should be ignored. Subscribe today to learn what is really going on with gold, and to see how we are actually trading gold, silver, and the precious-metals stocks.

The bottom line is gold futures activity as chronicled in the CFTC's Commitments of Traders Report is often misunderstood. A minority of analysts choose to interpret facts about week-to-week developments out of the illuminating context of bull-to-date behavior in similar situations. Thus their interpretations of this complex report are often misleading. And sadly many newer traders are swayed by this shoddy analysis.

It is critical to remember gold futures are a zero-sum game. For every short, there is an offsetting long. So if the feared commercial hedgers' net-short position is surging and hitting records, then so too are speculators' net-long positions. And as we've seen since 2001, as this gold bull powers higher interest in gold futures continues to grow. So we'll continue to see many new gold CoT records in the years to come.