It's shared responsibility, but IMHO there is nothing you can do to hold the tempo back if the drummer decides to run away with it, and refuses to accept any responsibility for time.
Ironically what you need to do is the opposite of what you will probably do. If the drummer thinks you are dragging, he/she will push the beat ahead and this will often cause the tempo to increase. You would think that if you pushed the beat the drummer would perceive you as rushing and pull back, but usually they just go with. If this is your situation there is nothing you can do.
What this seems to indicate to me is many drummers like to play tunes at a faster tempo than they are counted off at. There have been many times where I was playing a bassline in 4 like Johnny Be Good or Roll Over Beethoven that eventually get so fast that I have to start playing in 2

. And of course the drummer really thought it was grooving.