I too have thought about this a lot. I tend to be somewhat minimalist by nature. I also grew up in the era when most had 1 instrument, 1 amp.
I could do with just one bass, but I see having more than one as just having evolved along with the prolification and growth of electric/electronic music. Styles of music as well as types of basses have multiplied, so it's a lot more complex issue than just what you might be able to get by with.
I have fretted, fretless, long and short scales all, passive and active for justifiable reasons in that one can not duplicate the others traits. I did give up a five string and convinced myself that if I needed a low b, i can string one of my fours BEAD.
For myself, there is no conscience issue such as family, income, priorities, etc. but I can understand how you feel that way.
Believing strongly in balance, it seems to me things are not always black and white, and reasonableness allows us a place between the two extremes, more often than not.