Not my experience. But I like to ride the edge between clean and driving and bigger amps would often be louder than I wanted.
If headroom is yer thing they yes bigger = better.
I get most of my sound by the way I play the bass, I like an amp that doesn't get more distorted rather than louder at a certain point. You can get a nice overdrive out of an SVT but as you say they're too loud at that point usually.