The answer depends on a number of factors, with the main one being what type of music are you playing and what are you trying to accomplish? Beyond that it comes down to ergonomics, personal preference, etc.
For example, if you're playing in a classic rock cover band, a 4 will likely do the job as it originally did the job. Could you play a 6? Sure. Would it sound good? Depends on how you play it. If you're playing prog or modern country, a 5 will likely do the job as it originally did the job. Could you play a 4? Sure. Would it sound good? Depends on how you play it. Would it sound "right"? Depends on the context.
After playing 4 exclusively, then 5 exclusively, the swapping back and forth, then experimenting with 6, I had moved back to 4-string due mostly to the fact that I have a pair of instruments that are so good, I can work around the range limitation for certain material (though one is 24-fret and has a D-tuner). Then I got a call for doing music for a live-streamed show where I was going to be responsible for a lot of the "color" (doing as a trio, lots of improv). I bought another 6'er for that, as I knew that I was going to want to be able to quickly create odd rumbles (perfect for a B-string) and also get up in to guitar range (no guitar on the gig).
Bonus is that while working up my 6-string chops for the show, I'm finding some other tasteful uses in other applications (e.g. Brazilian pop band) where the extended range can be used to nice effect.
Long way of saying what I said at the beginning - what type of music are you playing, what are you trying to accomplish, and can you do it with the bass in your hands?