Personally, for me, its not that hard. And I find uses for the low B string all the time. On a 6, I rarely used the C string and it made the reach to the E and B strings kinda tough for me. But, 5 is not that much tougher than 4 and its nice to those low notes when you want them.
Now, one of the issues that lots of people have when making the switch is the string spacing. Most 5-string basses have the strings closer together than on a 4-string. The reason they do this is so that the neck of a 5-string is not really wide. But, it can mess up your feel if your not used to it. Of course, some people actually prefer the tighter spacing, so you never know, you might love it. Schecter, Warwick, MIM Fender, and Ibanez all have really tight string spacing (16.5 mm compared to 19mm on a normal 4-string) on their 5ers. Try a few of those out and see how they feel to you. For comparison, the Squier Deluxe Active V has 19mm spacing, just like a 4, so the neck on those is really wide. Most GCs carry them so you can play those and compare it to the ones mentioned earlier with tighter spacing.
Even if you decide to stick with a 4, you will have ventured outside your comfort zone by trying out some 5ers. That alone makes it a worthwhile endeavor IMO.