Hiya, Wuggles...
Ok, this could be a number of things.
You say it's ok when you actually fret? So it isn't finger strength causing in the problem (the most common problem for beginners).
If it's doing it when you take your finger OFF the string, I would say it has something to do with the set-up of the bass. Is it an old bass? Second hand? Are you playing it through an amp? How far are the strings from your fretboard? A few millimeters? Centimetres? Inches?
Look down the neck of your bass. Is it bent at all? Even a slight bend in the neck can cause buzz...