| Leaving it in would make the notes where the first few frets were odd to play. You're pressing the string further down, which increases the tension, marginally over long distances but closer to the zero-fret it'd be worse, though you could correct for it with your ear.
However, on a similar note, taking it out will extend the scale length by a little bit and they way its extended (lengthening the distance to the first fret) means that you can't fix the intonatoin using the bridge, only in your fretting. This means that your lines won't be usable, which isn't bad (cause you'll rely on your ear) but they'll throw you around.
I'd leave it in I think but see if I can take the fret down, file it or whatever (you'd possibly also have to take the nut down somewhat as well, maybe). I've never actually seen a bass with a zero fret in person, so I'm not the most experienced on the matter.
Josh D | 
