Some of what frustrates clients is a fixable communication problem. Some of it is not, and the difference is worth being clear about.
Most of what goes wrong is expectations that were never made explicit. Say plainly what is not working: how often you expect to hear from them, what you want done differently, what you need in the next two weeks.
If it improves, you have saved yourself starting over with someone who does not know your situation. If it does not, you have a clear basis for the conversation with the brokerage. How cancellation works →
If you are considering this because you already have someone else in mind, be straightforward about it. Everyone in this industry has been on both sides of that conversation, and handled openly it is far less awkward than it feels.