Is this just an issue with the tubes?
It is impossible to know. Loss of volume can also be the result of intermittency somewhere, like worn out / dirty potentiometers or switches in things like FX loop jacks. Practically there are dozens of other scenarios besides bad tubes. ...like the aforementioned battery.
But bad tube is definitely easy to rule out, right?
They don't appear to be glowing
This may not mean anything. Not all tubes glow very brightly to begin with and much of it has to do with the internal construction of the tube. In some tubes the filaments are almost hidden from sight (and such tubes have almost no discernible glow at all) while some tubes have their filaments much more visible. Do note that many tube amps have LEDs or other lamps to illuminate the tubes to give a more prominent impression of glowing. The designers do this because today people expect to see distinctive glow in tubes and the glow of the filaments alone isn't usually bright enough for their tastes. If it doesn't glow like sun people think it doesn't work properly, despite the amp actually working and emitting sound as usual.
Both these preamp tubes operate normally. Their internal architecture is different though, which means the other tube seems not to glow at all in comparison to the other. But this just shows how little importance the magnitude of the filament glow actually has. Both tubes in this case work just as they are supposed to.
Obviously you are more interested in whether the glow has reduced from what it was earlier or not, as that would be the only real pointer that a tube has started to malfunction in some way.