I'd suspect it's just you not being precise enough with the fretting finger.
You shouldn't need to apply a whole lot of pressure if your bass is set up nicely, but finger placement just behind the fret should help. The further behind the fret you press, the more chance the string will have to rattle against from it as it vibrates.
It *could* still be a setup issue, I suppose - but if its something that gets worse as you play faster kinda suggests that it's more of a technique issue than a setup one.
Wouldn't hurt to learn to check and set the action anyway though - have a read
here for pretty much all the info you're likely to need and more.