Week 51 Day 1: Trying to Beat the Market: The Professional Failure Rate

The S&P Dow Jones SPIVA (S&P Indices Versus Active) Scorecard publishes the definitive comparison every year. Over the 15 years ending December 2023: 92.2% of U.S. large-cap funds underperformed the S&P 500. 95.3% of U.S. mid-cap funds underperformed the S&P MidCap 400. 97.8% of U.S. small-cap funds underperformed the S&P SmallCap 600. And the few funds that beat the index over one period almost never repeat. SPIVA's 'persistence scorecard' shows that of the funds in the top quartile in any 5-year period, fewer than 3% remain in the top quartile for the next 5 years. The winning fund managers are not skillful -- they are lucky. And luck does not persist.

Why do professionals fail? It is not because they are stupid -- it is because the market is efficient enough to make consistent outperformance nearly impossible after costs. The mechanisms: (1) Management fees. The average active fund charges 0.6-1.0% per year. The average index fund charges 0.03-0.10%. That 0.5-0.9% headwind compounds devastatingly over decades. (2) Trading costs. Active managers trade frequently, incurring bid-ask spreads, market impact costs, and commissions. These hidden costs add another 0.2-0.5% per year. (3) Cash drag. Active funds hold 3-10% in cash for redemptions. Cash earns less than stocks over time. (4) Tax inefficiency. Frequent trading generates short-term capital gains taxed at ordinary income rates. (5) Reversion to the mean. Any outperformance attracts new money, making the fund larger and harder to manage. The fund that beat the market when it was small struggles to repeat when it is 10x larger. The implication is liberating, not depressing: you do not need to find the needle in the haystack. Buy the haystack (a total market index fund) and you own all the winners automatically. The time you would have spent researching stocks and funds is better spent on your career, your health, your family, or anything else that brings you joy. Investing should be boring. If it is exciting, you are doing it wrong.

The intellectual foundation for indexing was laid by three independent lines of research: (1) Samuelson (1965) proved that in an informationally efficient market, prices follow a random walk and future returns are unpredictable. (2) Fama (1970) formalized the Efficient Market Hypothesis (EMH), showing that all publicly available information is rapidly incorporated into stock prices, leaving no systematic profit opportunity for active managers. (3) Sharpe (1991) proved arithmetically that the average dollar of active management must underperform the average dollar of passive management by the cost of active management -- this is true by definition, not by assumption, and holds regardless of whether markets are efficient or inefficient. The SPIVA data confirms Sharpe's arithmetic: the average active fund underperforms by approximately the amount of its fee. The minority that outperform in any given period is statistically consistent with chance (Fama and French, 2010, showed that the distribution of alpha among mutual funds is indistinguishable from what would be expected under zero true skill). Barras, Scaillet, and Wermers (2010) used false discovery rate methodology to separate skill from luck among 2,076 actively managed U.S. equity funds from 1975 to 2006. Their finding: 75.4% of funds had zero alpha (pure luck/noise), 24.0% had significantly negative alpha (destroyed value after fees), and only 0.6% -- approximately 12 out of 2,076 funds -- showed statistically significant positive alpha. Even this 0.6% could not be identified prospectively. The practical conclusion is unambiguous: the probability that any individual investor, using any selection methodology, will identify a persistently market-beating fund before the fact is negligibly small.