Funny, I asked the same question years ago when I started on bass and now I think I can safely give an answer.
You have to examine what is exactly this desire to hear fast. I'm allowed to rephrase it, it's actually to be came to come up with phrases that fit a fast tempo. You have to invent new phrases quickly to match the tempo you're asked to play at. To invent anything, you have to start from somewhere. You're trying to imagine sounds quicker, and the only real way to do that is to feed that imagination. IMO, you can practice scales and such til you're blue in the face and your imagination won't really grow or speed up. The only real way to feed it is to play melodies and study recorded solos through learning by ear and picking it apart phrase by phrase. Once you develop an ability to conjure vocabulary effortlessly, then things get easier to speed up.
But also there are a couple key things:
1) You have to have the melody memorized. The melody itself is a source for inspiration.
2) You need to have the changes not only memorized but also internalized so that you're not fighting to keep track of where you are in the tune.
Usually if I have 1 & 2 down, and over the years worked on developing my own language, then I can START working on playing fast in a coherent way with some level of effectiveness. The clarity comes alot easier. Takes lots of work and I'm certainly not there yet.
Take a look at Ed's thread about REALLY LEARNING A TUNE here: REALLY Learning a tune
IMO, that's prob one of the fastest ways to start getting into playing everything the way you want. It's a long drawn out process tho.