When you get a chance to hear 100 or more of the same instrument, it gets pretty easy to isolate the differences a fingerboard will make based on your understanding of the average maple neck. Of course, that is something that usually only builders get to do, or possibly store owners.
Between my repair, building, and playing experience I must've played at least a couple of hundred each P basses and J basses over the last 40+ years and even if I played all those necks on the same body I wouldn't venture to try to state anything definitive. I've heard plenty of crossover between necks with maple and rosewood fingerboards.
The only valid conclusion that can be drawn is that different necks can sound different.