It's a tricky situation because it's a subjective issue. It's a matter of taste, and that makes things harder to justify.
If the guy was blowing off gigs and rehearsals, unable to perform the material, compromising the band by engaging in illegal activities, etc. then it's easy to fire someone. You have a justifiable reason that's easy for folks to get behind. But, taste is a subjective thing and one of the reasons you have millions of people that hate Justin Beiber and millions who love him.
You say he can't play a Latin beat. Do you play Sambas and Bossas? If not, who cares?
Do you want to play Latin material in the future? Well that's a matter of taste. If that's the direction the band wants to go, then it's a real issue. If not, then let it go. I work with lots of drummers who have difficulty with certain beats. I just don't use them on gigs where I need that groove.
I'm assuming the drummer was good enough when he became a member of the band, so when you say he's holding back your sound, what is the issue or effect? Is he unable to play an arrangement? If so, then it's less subjective. But does he know the arrangement? Have you guys talked to him about his dynamics? If there a musical direction the band wants to take that's different from what the drummer can deliver, that's one thing. If there's a musical direction that you want to go in, but the drummer or the rest of the band doesn't then that's another thing.
IME, IMO, you can always find someone in the audience to justify your position. Even the crappiest singers and musicians can find someone in the audience who thinks they are great. Also, I wouldn't let the feedback from a single gig affect the direction of the band. If it's a trend, that's a different matter. But, a single gig could be an exception. One of my best nights was when I fronted a band and sang all night. Floor was packed, crowd stayed until the end, bar sold well, line out the door to get in, and musically we were 'on'. But, I wasn't going to justify firing the singer who couldn't make the gig and making me the lead because of that night. That's because I know I have had more nights where I was blah or average. Thus, I tend to avoid gigs where I have to sing all night even though I had a few gigs where I really lit it up. The trends related to my singing were that it was mediocre, while the exceptions were when my singing appeared great. I use the word 'appeared' because I don't think I was singing that much better, just better received.