A new survey from Spectrem Group finds that affluent investors – particularly millionaires – are becoming more and more pessimistic about the U.S. economy’s erratic recovery despite the Dow's strong start to the year.

Spectrem, a Chicago-based strategic consulting firm specializing in the affluent and retirement markets, reports that its Affluent Investor Confidence Index plunged 7 points in May to a -5 rating, down from a rating of 2 in April and a 5 score recorded in March.

Meanwhile, the firm’s Millionaire Investor Confidence Index fell 5 points to a 3 score, the largest one-month decline in almost a year.

“Affluent investors continued in May to remain on the sidelines,” George H. Walper, Jr., president of Spectrem Group, said in a statement. “The ongoing European Zone crisis and its impact on the global economy is the news story that is most affecting their economic outlook.”

The Millionaire Investor Confidence Index recorded scores of 10 and 8 in March and April, respectively.

In April, the Dow managed to squeak out a tiny 1.59-point gain for the month after soaring up 8.1% in the first quarter.

Larry Barrett writes for Financial Planning.