The first thing to do is to check your contract with the agent. If you had a sole agency period, there will be a time at the beginning of the contract when you are supposed to stay with them. If you are out of contract, there would not usually be a charge anyway.
Having said that, if the customer service has not been good, then they will be in breach of their obligations to you, which could also terminate the contract.
However, this could lead to complications if they don’t accept your point of view. You should start by talking to them about it. You may find that there are reasons they haven’t done what you expected, or that they buck their ideas up.
Failing that, if you are under contract, explain in writing what the problems are and give them a set time to remedy.