Another thought occurs to me here that may be interfering with your difficulties fretting.
1. The action on your bass strings may too high for you. If your strings are too high off the fretboard, they are much harder to fret. Some bass players actually prefer high action, but not every bass player is comfortable with it.
2. You may be using strings that are too heavy a guage for you. Lighter guage strings are easier to fret and to pick also.
3. Your fretboard may be too wide for your hand. If that is the case, the wide fretboard will slow down your fretting, though not have so much impact on your picking. I had one bass that I eventaully sold which had very wide frets. It took a much wider stretch of my fingers to cover four frets than on my Tobias which is much more comfortable for me.
4. Your fingers will be able to stretch more with time and consistent practice. Helpful drills can increase your fretting speed. Also fretboard familiarization increases speed. Often just that split second hesitation of thinking where to put your fingers next will slow you down.
5. Cross string drills both for fretting and picking help increase speed. You do need to develop facility with string crossing. Don't be one of those bassists who plays everything on just the E string.
I think it is Michael Manring who has a drill for beginners in which you play open strings--do not fret, but you pick on the E string twice; then pick on the E string; then move to the A string and pick that. Then you pick on the E string, then move to the D string and pick that. Then you pick on the E string and then move to the G string and pick that.
When you can do that with proficiency, you can start mixing up the drill. For example. Pick on the A string, then pick on the G string, then pick on the D string, etc. There are dozens of variations.
Once you have picking confidence, then fret notes on each string instead of playing open strings while you pick.
Another thing I suggest is using a metronome for these drills. Start REALLY slow. Build up speed gradually. Accuracy is more important than speed until you gain confidence and agility.
