(Bloomberg) -- At first everything’s great. A client and advisor talk all the time, set life goals together, exchange notes. One day the client notices the conversations have gotten shorter, the notes less frequent. Calls go unanswered. Maybe the client and advisor aren’t such a great fit after all.

The problem is this advisor manages the client's life savings.

“The good and the bad of the financial advice industry is that it’s so deeply rooted in the relationships that form between advisors and their clients,” said Michael Kitces, a Financial Planning contributing writer, partner at Pinnacle Advisory Group in Columbia, Md., and writer of Nerd's Eye View, a widely-read industry blog. "It feels like (the client is) firing a friend. And most human beings don’t like confrontation with other human beings.”

In one classic "bad" scenario, Kitces said, the advisor “seemed really nice and friendly, and (the client) moved a little bit of money” into a simple financial product, only to find the advisor becoming “increasingly aggressive” and pushing the client into something “more complex, and (they're nervous they) don’t really understand it.” Kitces's advice: Don't wait for the investment to blow up in their face.

"Sometimes (the client's) intuition is trying to tell (them) something about the relationship," he said. "Clients even acknowledge a point in the middle when they knew something wasn't right with the dynamic but didn't do anything because it's awkward."

One man who was trying to settle a family estate approached Lisa LaMarche, a partner at Milestone Wealth Advisors in Greenville, Del., and said he was "getting horrible service from his advisor's team. They wouldn't return his phone calls, wouldn't turn over the money" to the beneficiaries, LaMarche said. She took part in an exit call with the man and his advisor—whom she found polite and cooperative.

LACKING TRUST

However the break came about, she said, "if a client feels that their advisor is unavailable or is providing explanations that do not make sense to them, they develop a lack of trust in that person." Her view is that "when it comes to client experience, perception is reality."

Nola Kulig, a Longmeadow, Mass., financial advisor, helped prep a woman to explain why she was leaving her advisor, whom she described as more of an insurance salesman. Kulig suggested the client tell him she had decided to take "a more holistic approach" to her finances and was reviewing everything in her portfolio.

If an advisor has lied to a client or misrepresented fees or investments, as in this nasty cavalcade, the client ends it, period. If they've simply fallen out of love after a long relationship, or have a nagging feeling they could do better, it can be more like the Seinfeld episode in which Jerry agonizes over breaking up with his barber.

But, as with LaMarche's example, it isn't always black and white. A study by the White House Council of Economic Advisers estimates that conflicted advice, short of outright lying, may cost retirement savers as much as $17 billion a year. Many of Kulig's clients come to her from big brokerage firms wondering whether they're getting the best fund or just the one that best compensates the salesperson, Kulig said.

"It's hard for someone to say, I trust you, but I don't trust your firm or how you're compensated," she said.

One woman on a personal finance message board said she has assumed her advisor was looking out for her best interests; she had worked with him for 16 years, and he was "really nice" and a strong supporter of his church. When she raised the "unreal fees" she was shouldering and asked him if there were cheaper products, he said she was paying for the benefit of actively managed funds and their superior performance. An astonished commenter told her to dump the guy and reassured her that "he'll survive."

"If you're worried about your advisor's reaction, keep in mind that he's enjoyed 16 years of commissions, loads, 12b-1 fees, wrap fees, etc.," the commenter said. "He's done OK."

THE MARRIAGE ENDS

Ending it can be especially hard if everything's been fine. Debora Killeen worked happily from afar for many years with an Ameriprise advisor in the Bay Area after she moved from California to France. They enjoyed each other—she was six months older, and he called her Big Sis. When she returned to the U.S., living in Sacramento, she wanted to take a more active role in her finances, with access to a local office where she could stop by with questions. He was two hours away.

"The long-distance relationship wouldn't work for me," Killeen said. She began meeting with a woman at an Edward Jones office near her home, enjoying an education in money management.

"Emotionally, it was like getting a divorce," she said. She composed an e-mail to her Ameriprise advisor explaining why she was moving her account and called him right after hitting send. He didn't answer immediately, but when he called back they talked for 15 minutes.

"Like my friend who is a lawyer said, it's business, and he knows it's business," Killeen said. "He was very encouraging and was like, 'Take care of yourself, Big Sis.' " She was "happy that I could move on." 

Advisors can be leery of clients, too. Austin Frye, an advisor in Miami, was introduced to a man at a Christmas party who was very unhappy with his advisor because his portfolio was down for the year. The guy had had a few drinks, and Frye remembers the "veins bulging out on his neck."

"I'm thinking, 'Hey, the market's down, and that's why you're down." Frye said. "I don't want you as a client."

WEIGHTY CONCERNS

Early in his career, Frye took on a client who admitted he was ditching his financial advisor because the guy had gained a lot of weight and the client feared it showed a lack of control. Two years later, Frye got a note from the client saying he was going back to his old advisor, and called to ask why. “He lost weight,” the man said.

How your advisor reacts when you announce you're leaving will tell you a lot, if belatedly. "If an advisor goes into sales-pitch mode at that point, then the client should know that they have a salesman and not a money manager," said Joseph Krier, president of Krier Wealth Management in Jacksonville, Fla.

"Just give me another chance. I promise it will be better," Pinnacle's Kitces recited. "I've done all this work, how can you fire me now? We've worked together for so long."

So it's not surprising that many clients fire their advisors by e-mail. There's nothing wrong with that, to Kitces, and it provides documentation in case you need it down the road.

Two people can leave the same conversation with two very different ideas of what they achieved in it. If the client softens the blow too much when they and their adviser talk, they might not realize the client fired them. Clients learn this when, three weeks later, they make a bad trade on your account. Then they've got a misconduct issue to deal with. And a brand new relationship with their old flame.

Read more: