As a mortgage broker during the manic years of the housing boom, I witnessed reckless financial practices on a wide scale. As a result, I was not surprised by the "robo-signing" mess that now threatens the mortgage sector. Unfortunately, the scandal is only a small tip of the iceberg that threatens to take down the entire US banking system.

The "too big to fail" (TBTF) banks that acted as middle men in the mortgage machine knew that the mortgage-backed securities (MBS) they packaged and sold to investors didn't meet the standards they claimed. In essence, MBS buyers were sold Ferraris but took delivery of PT Cruisers. Because of these material misrepresentations, TBTF banks could be forced to repurchase hundreds of billions of MBS that they sold to investors. Since they don't have that kind of cash lying around, it's likely they will turn to their federal benefactors for another bailout.

If the Treasury is solvent enough to offer such a bailout, there might never be a proper investigation of how the mortgage market blew up. As a former industry insider, I hope I can shed some light.

Historically, real estate prices in the United States were driven by supply and demand, and generally tracked the rate of inflation. Securitization changed that. Mortgages no longer remained with the lender extending the loan, but were sold to an investment bank that would, in turn, create MBS and sell them to investors. During the housing bubble, trillions of dollars of these securities were churned out. When banks found themselves with loans that did not conform to minimum requirements for securitization - rules meant to protect MBS investors from the risk of widespread defaults - their response was to sprinkle these "subprime" loans into other MBS bundles. Auditors only sampled about 5-10% of the loans making up any one MBS, so if a particularly egregious loan was rejected, it could simply be shuffled into another MBS. In the rare case that the loan was spotted a second time, it was shuffled again.

Meanwhile, the banks largely didn't share the poor audit results with potential MBS investors. They were savvy enough, however, to use the poor showing to negotiate lower prices from wholesalers who sold them raw mortgage loans. This is like a grocery store demanding refunds from farmers who supply E. coli-tainted spinach, but selling the produce to consumers anyway - with no warning, and at full price!

Emboldened by recent auditor testimony of such practices, and citing failure to properly service mortgages according to signed contracts, a group of investors has filed suit to force the banks to repurchase their toxic MBS. In response, the banks have publicly stated that servicing problems are "contained" and present no broad systemic risk, even while it becomes increasingly clear their problems extend far beyond technical breaches of procedure. As more information concerning the extent of potential bank fraud comes to light, more such lawsuits are a certainty. The situation presents lawyers with a once-in-a-lifetime opportunity.

As a result, I expect uncertainty and headline risk to haunt US banking stocks for some time. Housing prices will likely fall much further as foreclosure sales are delayed and homeowners increasingly decide that strategic default is their best decision. As I mentioned, I don't believe the banks can withstand this tidal wave without a flood wall of federal funds.

Which brings us back to the great question of this economic crisis: should we consider these banks "too big to fail"? Or, rather, is it time that they must fail?

As recently as April 2006, I was still an optimistic mortgage broker. If I had heard of Peter Schiff or the Austrian School of economics, I may have thought twice about signing a three-year commercial lease and spending thousands on office equipment. But, like most people at that time, I couldn't see how real estate prices could ever fall. By 2007, my business was floundering. As cracks in the housing market started to show, small brokers were vilified in the media and squeezed out of the market as banks raised their underwriting guidelines to eliminate "unscrupulous" external loan brokers. So, small shops like mine closed one after another. Meanwhile, the big banks continued to purchase and underwrite billions in loans, even while they were aware of declining underwriting standards and fraud within their ranks. I vividly recall that as my business was drying up, my only option to continue my trade was to return to a big bank. They were the only ones still making money as the market collapsed.

Fortunately, my ultimate perception (aided by Peter's analysis) that the housing market would not improve any time soon led me to make a pragmatic decision: get out and move on. I took the losses, and I lived to fight another day. Nobody bailed me out, and nobody is bailing out the individual homeowners across the country faced with upside-down mortgages. The much touted Homeowner Assistance Modification Program (HAMP) has proven to be a bust, a byzantine maze of red tape and frustration awaiting any homeowner who seeks its aid.

An increasing number of homeowners are making their own pragmatic decision: stop paying the mortgage, save money, stay in the home as long as possible, and, eventually, get out and move on. The moral hazard created by the Wall Street bailouts has encouraged business-as-usual from the TBTF banks, who have largely refused to restructure underwater mortgages, and has promoted more foreclosures as homeowners orchestrate their own private rescue packages. Over the next few years, I imagine it will be increasingly clear to Americans that when it comes to the big banks, it's time to cut our losses, get out, and move on. Take it from someone who has done so, and never looked back.

For in-depth analysis of this and other investment topics, subscribe to Peter Schiff's Global Investor Newsletter. Click here for your free subscription.

Click here to download Peter's latest Special Report: My Five Favorite Gold & Silver Mining Stocks.

Be sure to pick up a copy of Peter Schiff's just-released economic fable, How an Economy Grows and Why It Crashes. Click here to learn more and order.