Forty-seven percent of Americans say they've either canceled or pared back their summer vacation plans this summer due to surging gas prices and another 20% said rising air fares have forced them to modify their travel itineraries to save money.

The RBC Consumer Outlook Index for May fell to 42.9, down 1.9 points from April's 44.8 score.

"With gasoline price increases showing little sign of abating, this month we asked once again how this impacts discretionary spending," Tom Porcelli, chief U.S. economist at RBC Capital Markets, said in the report. "The results highlight the psychological element involved with rising gasoline prices, as people revise their pain threshold according to the reality of what is happening at the pump."

With gas prices at or above $4 a gallon throughout most of the country and oil prices still well above $100 a barrel, consumers are being hit by the double whammy of higher fuel prices for their vehicles and higher prices for all types of goods as companies raise their prices to cover the additional operating expenses.

"This suggests that, despite what is likely to be a continued drift higher in gasoline prices given supply and demand dynamics, spending is very likely to hold up," Porcelli added.

Three-in-10 Americans surveyed between April 28 and May 1 said they have curtailed their discretionary spending as a direct result of higher prices at the pump. Another 41% said they would definitely reduce their consumption if -- or maybe when -- gas prices eclipse $4.50 a gallon.

Back in March, a similar survey found that half of consumers would begin cutting back their gas consumption when prices exceeding $4 a gallon.

RBC's Consumer Outlook Index is comprised of responses to four different sub-indices. For May, the Expectations sub-index fell 1.3 points to 53.2, the lowest point so far this year while the Current Conditions sub-index shed 3 points to 32.9. Seventy percent of the 1,000 respondents surveyed said the country and economy are on the wrong track.

The Investments sub-index tumbled 3 points to 36.5 in May, even though the stock market has performed quite well in the past year. Thirty-four percent of consumers polled said they think it's a bad time to invest in the market and 50% are generally unsure about investing at this juncture.

Only 28% of respondents said they know of someone in their immediate circle of friends and family who is concerned about losing his or her job and only 40% said they know of someone who has lost their job in the past six months. Still, the Jobs sub-index shed 1 point to 50.8.

"Consumer confidence remains fragile, as evidenced by the broad-based decline in the RBC Consumer Outlook Index - indeed, all sub-indices posted declines this month," Porcelli said. "But what a consumer says can sometimes be at odds with what a consumer does."

 "Confidence fell, but weekly measures of consumer spending remain fairly buoyant and, from our perspective, the second quarter is looking very much like the first quarter," he added. "In other words, consumption is not losing momentum."

It's worth noting this survey took place before an elite U.S. special forces unit killed Al Qaeda leader Osama bin Laden in Pakistan and therefore doesn't reflect any potential changes in overall U.S. consumer sentiment this turn of events may have elicited.