Market-watchers hang on just about every word that Warren Buffett says, and for good reason: The legendary investor turned an initial investment in a small New England textile company into a multibillion-dollar portfolio. But as is the case with many great dispensers of advice, Buffett suggests you do what he says, not what he does.

Buffett is an active investor, picking and choosing individual stocks he sees as attractive investments. If you're looking to emulate Buffett, you might buy your own portfolio of individual stocks. But Buffett says this isn't actually a good idea.

That's because, unlike the Oracle of Omaha, you probably don't devote your waking hours to diving into investment research. You could take another active approach and invest in actively managed mutual funds helmed by managers who build portfolios designed to outperform the market. But Buffett advises against that, too.

"Active investing as a whole is certain to lead to worse-than-average results," Buffett said in a 2016 interview with CNBC.

Video by Courtney Stith

Instead, he says, investors should buy passive investments, which seek to replicate the performance of a market index. By holding, say, a mutual fund or exchange-traded fund that tracks the S&P 500, you'll likely come out ahead of your peers over time, Buffett's thinking goes.

As is often the case, the numbers back up Buffett's argument. Read on for three reasons it's smart to favor passive investments over active strategies.

Beating the market is hard, even for the pros

If you own an active mutual fund, you may see market-beating results over the short term. But over time, it's hard for mutual fund managers to keep up. In the 15-year time span that ended in 2019, only 37% of mutual funds invested in U.S. stocks outstripped their benchmark index in a given year, on average, according to Morningstar.

But that's just mutual fund managers. What about the people who manage hedge funds, investment vehicles for the uber-rich? It turns out, they're not much better, as Buffett himself demonstrated.

In 2008, the Oracle challenged the hedge fund industry to outperform the S&P 500 over the subsequent decade (investor fees included – more on that shortly). Asset manager Protégé Partners accepted the $1 million dollar bet, pitting a portfolio of hedge funds against an S&P 500 ETF. They conceded in 2017, before the final results were in.

Outsmarting the market is hard for you, too

Even if you hold passive funds, you may be tempted to take an active approach in your portfolio, selling in and out of the market based on news that affects stock prices. You're almost sure to lose out on potential returns this way, too. A 2020 study from research firm Dalbar found that the average stock investor has earned a 7.8% annualized return in the 25 years ended in 2019, compared to a 10% return in the S&P 500. The culprit: poor market timing, with the typical investor holding funds for an average of just 4.5 years.

Video by Stephen Parkhurst

Faced with questions about current events at the 2018 Berkshire Hathaway shareholder meeting, Buffett walked his investors through all of the potentially market-disrupting events in the U.S. economy since 1942. "The best single thing you could have done on March 11, 1942, when I bought my first stock, was just buy an index fund, and never look at a headline, never think about stocks anymore," he told CNBC, recounting the meeting.

Had you invested $10,000 in such a fund at the time, you'd have ended up with $51 million, he pointed out.

The best reason to go passive: fees

You can't control how your investments will perform, but you can control what you pay for them. Because passive funds merely replicate the composition of an index, they are inexpensive to manage and charge low fees to the investor. Active funds typically come with higher fees to compensate a manager who trades more frequently in an effort to outrun the stock market.

According to the latest fee study from Morningstar, the average passive fund charges an expense ratio of 0.13%, compared with a 0.66% average fee among active funds. Hedge funds typically charge an exorbitant 2% fee, plus an eye-watering 20% of whatever the fund makes over a predefined return threshold. So it's easy to see why Buffett bet $1 million on an ETF charging 0.04% in fees.

But even fractions of a percent can make a big difference to your return over time. If you invested $10,000 in a fund returning 8% annualized and charging 0.66% in fees, after 50 years you'd end up with $337,000, having paid $32,000 in fees. Say you paid the same amount and earned the same return, but this time invested the way Buffett suggests – in the Vanguard S&P 500 ETF (VOO), which charges 0.03% in expenses. After a half century, you'd have $462,000, having paid less than $2,000 in fees.

 More from Grow: