@OldSchoolFlats ,
I play both by interval and by visualizing the notes on the finger board. My fluency varies somewhat depending on the key. I am a retired pro and the only reading I currently do is play changes and sometimes rework melodies from the Real Book (which usually entails reading in treble clef).
Reading is a skill that must be developed, and also maintained. If you play frequently in difficult keys, those keys will become easier. If you stop playing in those difficult keys, your competency level will drop. There was a time when I was very comfortable reading in Gb major. Probably not so good at it now, because I have not done it for years. If I am improvising over chord changes, my mind can sort of go on automatic because I know the patterns and have a decent ear. The idea is I use a variety of mental processes when I am playing. If one process fails, I may still play the right notes because another mental process kicks in to cover the gap.
Based on what you have written, my impression is you haven't fully explored the idea that there are multiple ways to play different lines. You might think of it as different hand positions, or different shift and finger patterns. Sorry, but if you want to be a fluent reader you can't rely on just one way to play a certain pattern.
What determines the specific pattern you use is where you are coming from and where you are going. For example, if you were to play just a minor 7th arpeggio from root position up, you would probably start with your index finger on the root and play 1 (root), 4 (minor 3rd), 3 (perfect 5th), 1 (minor 7th).
But this may not be the best pattern to use in the middle of a bass line. Instead you have to consider where you are coming from and where you are going. The line you are playing may set you up to start the arpeggio with the pinkie, so you might play 4 (root), 2 (minor 3rd), 1 (perfect 5th), 4 (minor 7th).
Both of these patterns allow you to play the minor 7th arpeggio with no shifts, and my guess is you will find starting on 1 to be easier. But there are times when it makes more sense to use the second pattern, so you need to know both.
Also you can play the pure minor scale (root to root) with both of these hand positions. The first position allows you to play from root to root with no shifts. The second position requires at least one shift to play the root at the top of the scale. Ideally you need to know the intervals between all fingers and the notes under all fingers with both hand positions.
To be a good sight reader you should be able to place a finger anywhere on the fretboard and name the note...of course this is a bit complicated because of the concept of enharmonics: Since C#=Db you have to figure out which note to call the fret, and that requires you to understand where the key center is. My point is your mental map of the hand positions changes depending upon the key center.
IMHO practicing modes can help expand your understanding of key centers and also open up your mind to different finger patterns. Try to work out a variety of finger patterns for the different scales and modes. Also, I suggest learning a variety of patterns for your 7 chords.
If you use one finger per fret, the box should be expanded from the four frets under your fingers, to the first fret immediately above and below your fingers. I think of this as 4 +/-1. Essentially you use micro pivots to expand the box by two frets. Scotts calls this 4+2: see video below.
It can be overwhelming and intimidating because there is so much to learn. In truth you could spend a life time studying music and still have plenty to learn when your time is up. I suggest focusing on enjoying the path to enlightenment, rather than expecting that you are going to reach some destination where you have learned all that you need to know. IMHO when you reach the point where you are no longer interested in learning, there is a good chance your motivation to play music will be greatly decreased.
