This guy says. . . . when I switched to 5 string basses, in 1996, I discovered bass playing plus. I love them. I find them easier to play than 4 strings basses, better sounding and way more flexible. It seems, reading TB posts, that many don't get on with 5 ers and some don't want to get on with them.
With the right frame of mind, changing formats is very easy. Of course it takes time to get used to the new geography and new note positions, but in comparison to the effort everyone had to put in to learning bass from scratch, switching is pretty straightforward.
I think what you say about the frame of mind is key.
I tried 5 strings on and off for 12 years before "I got it". The Lakland 5502 was sooo nice to play, the USA SUB5 sounded amazing... but I never dared playing them live for fear of making mistakes, so they eventually got sold. The last time I tried, however, it was a very different experience and as a result of it now I feel completely comfortable with any 5 string.
What was different?
This time I had no gigs, there was no 'pressure' about performing since I was playing just for myself. Another thing that helped A LOT was that since we were not gigging, we'd spend more time recording ourselves and we were writing new material... and I found the 5th string so useful! And lastly, the UB40 tribute band I had been playing with decided it was time to expand our repertoire and learn a bunch of new songs. Our singer is able to sing most of the songs in the original key, but he does struggle with a number of them and it's not unusual to decide to change the key to something else the singer is comfortable with, and after a couple rehearsals we change the key to something else again... On a 5 string I don't ever care, let's do a different key each time, I don't mind, I don't even need to think, which is where I want to be: when I'm thinking is when I make mistakes, my brain is faster/better than my physical ability
If playing a 5 string feels like a chore compared to a 4 string and you don't see a benefit... then it's going to be a lot harder, and maybe it's best to wait until you see a reason to do it.
If you're starting out on bass... it's going to feel equally difficult, so might as well go 5. A lot easier going 5->4 than 4->5 (based on my very statistically significant sample of one person, of course

)