As long as you don't hear it through the amp I don't think it's that big of a problem. Don't think it's the nut. Most likely bridge or neck or a combo of the two. Most basses will have a little buzz at some point in the neck, because nothing is perfectly straight. Some steps you can do to prevent fret buzz are as follows.
1) make sure your neck is straight or ever so slightly bowed. (and i mean ever so slightly). Good way to check the neck straightness is to fret the first and last fret on the E string and check the gap around the 7th fret. You want there to be a tiny tiny little gap in between. By tiny i mean 1/32nd of an inch.
2) Set the string heights to a point where you play at your normal attack and you don't get a buzz. (there may be a point on your neck where the is a buzz. You can tweak all day and maybe never get it completely out. You must compromise somewhere either by raising the action or dealing with the buzz)
you're probably getting a buzz because you're over compensating, because you're playing without an amp.
Just my 2 cents.