My question: is this just a matter of time and practice, and does muscle memory kick in for moving all around the pattern, or is this a common issue
Lots of good advice in this thread. Here is my 2 cents ...
Muscle memory is a good thing. Its great to have a lot of patterns that are embedded in your muscle memory to call on when you need them. Just playing a scale up and down though is not enough though. Play fragments of the scale (3 or 4 notes), play common patterns from the scale, play riffs from the scale.

.
.