The words of my bass instructor when I asked him this very question: "if you want to play six, go buy a six. Getting good on a four won't prepare you for a six, playing one will". I'm of the mind that you should play what makes you happy.
This is my thought as well - if you want to play 6 get a six. Sometimes the reason I pick up my 6 (and I ALWAYS pick up the 6 when not fretless) is because it just sounds so damned good
If you love the way the 6 looks, and you really want just play it a bit to ensure it isn't a dog and go for it. Like someone already said, enthusiasm for the instrument can definitely put it in your hands, and that's where you need it to be to become a better musician. Good Luck!