Wealthy investors apparently unnerved by rising oil prices and the potential for higher interest rates and inflation now say they are much less confident about the stock market and the broader economy than they were just one month ago.

Chicago-based Spectrem Group,  a strategic consulting firm specializing in the affluent and retirement markets, reports that its Spectrem Millionaire Investor Confidence Index (SMICI) in March fell to an aggregate score of 8, down 9 points from the downright bullish score of 17 recorded in February.

Despite the sharp decline in overall confidence among the wealthiest investors, the index is still perched at its second-highest level since December 2007 when it also attained a score of 8.

Likewise, Spectrem's Affluent Investor Confidence Index, which spans a broader sample group of investors with more than $500,000 in investable assets, shed 3 points in March to fall to a "neutral" rating of 1 -- not terribly auspicious but certainly much better than the -8 score recorded back in December.

"Millionaires’ investment confidence fell in March, erasing more than half the prior month’s gains and returning it to a neutral stance from mildly bullish," Spectrem Group President George Walper, Jr. said in the report."With the economy and political environment ranking as their top sources of concern, it appears America’s wealthiest investors have added a solid dose of caution to their investment outlook."

So what's keeping the rich up at night?

The survey found that the general economy and the U.S. and international political climate were cited as the most serious threats to investors reaching their household financial goals at 22% apiece. Unemployment (10%), market conditions (7%), inflation (5%) and health-related issues and costs (5%) also played a role in tamping down expectations among the 250 high net worth households queried