People may or may not hear a difference. Sometimes I think I do, other times I don't. It may be in my head. I've had a lot of basses over the years, and there is something unique in the sound of each of them.
This would definitely be true... EVERY bass sounds different, even ones that are spec'd exactly the same.
My guess is, those who hear a 'big' difference between the E string on a 4 string versus a 5 string are not taking this fact into consideration. It's amazing, for example, that when I had the opportunity to play a dozen or so identically spec'd Sadowsky Metro 5 nad 4 string basses (known for tonal consistency), that all the strings sounded a bit different on each one. Two pieces of wood, much less multiple pieces of wood glued and bolted together, will have quite an impact on the tone and response of a bass.
Attributing differences to the 'most visible' visual cue is called the 'vividness phenomenon' in my day gig business, and it is very common (e.g., this bass has a BadAss bridge and this other one that looks the same doesn't, and I like the one with the BadAss better, hence, I like basses with BadAss bridges)
Now, will a given PLAYER sound different on a 4 or a 5 string... of course. I can totally understand a producer or engineer wanting a four string bass if the 5th string will not be used, since many players (even very good ones) have trouble keeping the B string muted. This is a minor thing live (unless the player totally sucks), but a HUGE thing when laying down a track. Different issue.