Warren Buffett is 'not actually the greatest' investor of all time, Morgan Housel says. But he is the richest — here's his 'secret'

"His skill is investing, but his secret is time."

Warren Buffett, Chairman and CEO of Berkshire Hathaway.
David A. Grogan | CNBC

Warren Buffett's net worth is at least $96 billion, based on estimates from research firm Wealth-X. The finance icon's investing strategy is studied and touted by market-watchers, including Morgan Housel, author of the recent book "The Psychology of Money." Housel contends that while Buffett's success is partly due to being a skillful investor, his real advantage is how long he's been a successful investor.

"Buffett is the richest investor of all time," Housel writes. "But he's not actually the greatest — at least not when measured by average annual returns."

Here's the No. 1 reason Buffett was able to amass so much wealth, and how you can capitalize on it, according to Housel.

'His skill is investing, but his secret is time'

Buffett famously started investing when he was just 10 years old, and by the time the was 30, he had a net worth of $1 million ($9.3 million when adjusted for inflation), Housel writes. That early start let him harness the power of compound interest. Compound interest is the interest you earn on your money, plus the interest it's already accrued.

If Buffett had started investing in his 30s like most people do, his results would be much less impressive, Housel writes. Say his net worth was $25,000 when he was 30, and he earned a spectacular 22% annual investment returns. By the time Buffett turned 60, his net worth would have been $11.9 million.

"His skill is investing, but his secret is time," Housel writes.

"Buffett is the richest investor of all time. But he's not actually the greatest."
Morgan Housel
Author of "The Psychology of Money"

Others successful investors have gotten even higher annual investment returns than Buffett, he says. But because they invested for a shorter period of time, their net worth doesn't approach Buffett's.

Housel points to Jim Simons, a hedge fund manager and founder of Renaissance Technologies, who has seen annual returns approaching an average 66% since 1988. Even so, his net worth is closer to $21 billion.

"Why the difference if Simons is a better investor?" Housel writes. "Because Simons did not find his investment stride until he was 50 years old. He had less than half as many years to compound as Buffett."

How compound interest can help you build wealth

Compound interest, which Albert Einstein reportedly called the eighth wonder of the world, can help you when it comes to growing wealth and setting yourself up for retirement. The sooner you start, the more time your money has to grow. In fact, not starting sooner is the most common regret among investors, according to a Magnify Money survey.

If you start saving $5,000 a year for retirement at age 25, by age 65, you could have twice as much money as someone who waited until age 35 to start investing.

"Millionaires are made in their 20s and 30s, not their 50s and 60s," Fred Creutzer, president of Creutzer Financial Services told Grow. "If you wait until you're 50, you're never going to catch someone who started at a young age. When it comes to investing, the early bird always gets the worm."