Please remove whatever nut you have on your bass and replace it with a stick of soft pencil eraser. Tune, play. Sounds just like usually? No?
Have you actually done this? Have you seen it done?
I cant say nut material has no effect whatsoever on fretted notes. But I'm pretty confident that the difference between brass, bone, and any given hard synthetic would be too small to detect in any meaningful situation. And the differences among any of those hard materials (discernible or not) would surely be smaller than the difference between any hard material and something as soft as a pencil eraser.
If the test you described actually exists I would love to see the results. I'm not being sarcastic here, I'm actually curious. I bet if a stick was worked into a decently-cut wooden nut the fretted notes would sound fine. I have no idea about the pencil eraser though.