Top Five Wealth-Building Factors Among Millionaires

5. In the Right Place at the Right Time 5. In the Right Place at the Right Time

Being in the right place at the right time was the fifth most important factor for millionaire investors. That comes in just above frugality, which ranked just off the chart at number six.

4. Taking Risk 4. Taking Risk

This is where the high net worth and ultra-high net worth differ. Investors with over $25 million in assets ranked risk-taking as the fourth most important factor while millionaires with less than $25 million swapped risk taking with frugality, reporting that being conservative was more important.

"They [those ultra wealthy] don't view that you can save you way to becoming wealthy," said George Walper president of Spectrem Group. "There's no question that frugality has been magnified in the last four years since 2008, but even for these very wealthy it's still not in the top five critical factors for how they became wealthy."

That makes sense given the expectations of the $25 million plus investor, Spectrem said. More than 60% in that class look for annual returns of 9% or better and 11% strove for 25% returns, according to Spectrem.

3. Smart Investing 3. Smart Investing

Millionaires at all levels agreed that smart investing was the third most important factor. According to Walper, that has to do with all areas of investing from owning shares in a company to owning one's own company and choosing where to put the money.

2. Education 2. Education

Both the high net worth and ultra high net worth ranked their education as the second most important factor. That tends to be related to advance degrees, especially for doctors, lawyers and Ph.D.s, Walper said.

1. Hard Work 1. Hard Work

It's no secret formula; millionaires at all levels credited their wealth primarily to hard work. Only 32% of those surveyed by Spectrem Group attributed their wealth to inheritance.