Being willing to say "no, not with him" or "no, not at that venue" is the ONLY leverage you have to protect your rep as a serious player. As long as you are respectful and calm when discussing it, there should be no problems.
Unsolicited advice: don't talk about this in reference to a...