As a drummer and bassist, I hear your frustration. I have been in so many situations the other way around where everyone in the band is pulling a song and I follow along, only to have one lone person say it's all my fault the song is speeding up. So what do I do? I grab my metronome, put it in my ear and say "ok, let's play it to a metronome". I've done this a few times over the years, and as soon as I do, everyone mumbles... "ok, so maybe I am pulling the song a little".... it becomes real clear when someone is dictating the rhythm from a metronome and the other players are forced NOT to speed up.
What usually happens after that is we all concede that, indeed, the song sounds better when we speed up, and let's just accept that. My current band member has notes all over his charts not to speed up - apparently all his previous drummers would run with him when he picked up the tempos. Yet he would start all the songs!! We finally realized that if we didn't speed up a little, it just wouldn't sound right, but what he REALLY wanted was to slow things back down in the verses before speeding them up again a little in the choruses.
The point? The drummer will always be the fall guy for speed indescrepencies. A lot of drummers take offense to this. I get it, who wants to be told they're not doing their job right. If this guy is really off, he's probably aware he has a problem. Perhaps at a practice suggest running through a problem song with the whole band playing to a metronome and see what happens. As someone else mentioned here, not everyone is right or wrong - it might be that the songs need to speed up a little. But at least playing to a metronome will force the guy to focus on the beat and not the fills..... it might be worth a try.