After 3 years you should definitely be using your 4th finger regularly. Of course, the necessity of this depends upon the difficulty of the music.
One thing to watch out for is your finger flipping up far from the fretboard. Try to keep it under control.
A good exercise for that is to practice...
I used this set for playing both standard tuning and drop C, so it is fairly high tension, and not easily bendable. You may try a lower gauge strings, or downtune to make it easier.
It's physics... Due to the low frequency, the low notes do not have enough time to develop. There is a limit, and IMO bass sounds better when heard, even if not as fast as guitars or drums.
My best advice, if you're serious about getting good at bass, is to spend less time posting on TalkBass and more time practicing. TalkBass is not a substitute for interactive lessons. However, it is a great resource in addition to a disciplined course of self-study.