Don't worry about it too much because it'll come over time naturally. You will eventually develop an instinctive feel for distance between notes in a particular position on the neck. Large Position shifts usually require a quick glance down. Short position shifts can be worked on without looking.
I've done a lesson on smooth position shifts here:
http://www.talkingbass.net/smooth-position-shifts-for-bass-guitar/#more-244
Just work on these shorter moves back and forth without looking at the fretboard.
As for sight reading, it's imperative that you don't look at the fretboard unless shifting position. I always stress the importance of sight reading not just for the reading aspect but because you learn so much more at the same time. You learn the notes of the neck inside out and develop a really good feel for positions.
As a quick tip, place your hand in a one finger per fret position with your 1st finger at the 2nd fret. This gives you frets 2,3,4,5 under the fingers 1,2,3,4. Now this position gives you all the natural notes (diatonic to C Major) in one area apart from the low F at the 1st fret of the E string. Now you can assign certain fingers to certain notes.
E string : G 2nd finger. A 4th finger.
A string : B 1st finger. C 2nd finger. D 4th finger.
D string : E 1st finger. F 2nd finger. G 4th finger
G string : A 1st finger. B 3rd finger. C 4th finger.
Now it's simply a case of pressing down the appropriate finger when you see or need to play a certain note. Just stretch down or pivot for the F.
Once you become used to the notes in that area you can move on to accidentals.
Mark