I voted carrotwood not to have a good time, but because the other two choices did not reflect my belief.
Another poster stated that they swapped out necks often, and heard a difference. Of course, that's a neck/fretboard combo that's changed, not just the fretboard. That doesn't address your question, as we don't know how much the neck wood influenced the sound. I have never removed a fretboard after careful listening on a bass, and replaced it with a different type of wood. So I can't answer the question accurately. I've played a variety of basses with different fretboard woods, and many sounded different from each other, but there's no way to isolate how much of that difference is due to the fretboard wood. I'd venture to say that is the experience of the majority of bass players. Honestly, I don't believe any of us are in a position to definitively answer the question.