I would like to put your words under the microscope.
You used the word, feel, to talk about timing issues with the drummer.
What would be the reason for you to say that the drummer is slowing down?
Your sense of timing?
If so, based on what?
Based on some kind of "metronomic timing"? Coming from where?
(Are you acting like a metronome/click?
But you are against the metronome.)
Based on your "we-are-only-humans" (emotionally fluctuating) timing?
What if you are wrong about your timing?
What if your drummer thinks that you are speeding up?
Who is right?
From your words, "I have also never used a metronome for practicing", but, again from your words,
you are imposing your timing on the drummer/on the whole band.
If your drummer/or any drummer you play with
is constantly practicing with the metronome/click/timing device, can the drummer develop "close to perfect" - metronomic - timing?
Is it good or bad thing in your opinion?
You said, "Keeping time is up to myself and the drummer", but only you have the right to slow down/speed up your drummer.
Next.
I sense some confusion about "being able to play with a click".
Somebody at TB mentioned Steve Gadd's expression,
"The most important thing is to meet at Beat1."
I truly believe that his expression is more or less related to JAZZ MUSIC/JAZZ DRUMMING. (Maybe we could include some punk music also.)
I am more interested what's happening within just one bar(or within the rhythmic patter).
How are those rhythmic layers/rhythmic instrument's lines/notes interacting within just one bar? (or within the rhythmic pattern).
Is that one bar/rhythmic pattern "grooving"?
What happens rhythmically at the first/second/etc... cadence?
And so on....
From my personal experience, a timing device is very helpful and provides very good answers to my questions.
I can tell if a drummer is speeding up, or slowing down a song due to my internal clock. There is tolerance with this though. Speeding up, or slowing down a little is fine, but there's a point where an intervention must take place if it goes too far in either direction. When I play behind, or ahead of the beat to get the drummer back to a stable tempo, no one knows I'm doing that (not even the drummer!). It'a very subtle in that I first pull or push the drummer into not continuing to speed up, or slow down (stabilize), and then I slowly but surely bring them back to the tempo we atarted at. I have yet to have anyone in a band I've played with complain about my method. Why? Because they don't know I'm doing it.