Yep. You have to know what you're hearing. Just a general comment, not directed at you, just tagged to this post - reading music is not hard. The dot is on that space, it's an A, but since we're playing in Db, it's an Ab. Different dots mean different note lengths. That's about all there is to it. Then you practice and get better at it. "Fluent" at sight reading doesn't mean that you're just fluent at sight reading, it also means that you've improved as a player and can make connections and anticipate the upcoming notes.
I will say that bass is unique in the guitar family. We can do sight reading of notation far more easily than the skinny six stringers. Hey have to play 3,4,5 notes simultaneously and those notes can each be played at numerous spots on the fretboard so you're going from three notes that could be played six different ways to the next three note chord that could be played another six different ways and have to do a lot of mental math to pick the right chords played the right ways. With a lot of experience you can do it, but it's tough. That's why guitarists usually use tab for melodic lines and chord charts with rhythms for the chucka chucka parts.
With piano, on the other hand, a middle C is played at one spot only. A D3 at exactly one key. That's a perfect situation for standard notation.
Bass is somewhere in between. We usually only play one note at a time and can do standard notation with just a bit of practice. Skim the chart and look for any tricky parts, spend a few seconds planning those and you're ready to go.