Part of your problem may have to do with your choice of words.

If you're "plucking" the strings, you need to work on your technique first. Somebody posted this site the other day and I thought it was awesome
http://www.studybass.com/. You might find some help there. You should be sort of sliding your fingers across the strings from one to the next. I believe there's some good info on that this in the link.
Once you've got your technique right (or if it's already right) then the only thing that's going to bring the speed is patience and commitment. Trying to go as fast as you can until your hand cramps up sin't going to do it. Working really patiently with a metronome will. Trick is to find a nice comfortable tempo and just keep playing at it, for like 5, 10 minutes non stop. Take a break, then go up a notch. Repeat. Repeat again. Go backwards, go forwards. Take a long break, or hit it again the next day. It's hard to have faith in practicing slowly before you see that the results are really almost magical. The speed comes from muscle memory. Once your fingers go on automatic pilot you don't need your brain anymore and the speed just follows. Some of us never use our brains anyway.
Oh yeah - that website above has a nifty online metronome too.