Setup definitely affects sound - I don't keep any instruments without a straight neck and good fretwork, end of story... My typical setup would be described as medium low by most - it'll fret everything cleanly, but get a little buzz/growl when I dig in a bit...
It was set up by a luthier by the way who is a jazz bass enthusiast so I highly doubt set up is the issue.