Why are you trying to learn a song by ear? Just to improve your ear or do you actually need this song and can't find a transcription?
The reason I ask is, if you can't find a transcription and you're gigging on it I'd go ask for tab in the tab side

I'm right there with you, learning how to play by ear!
But, if you're just trying to learn... and you're beating your head against the wall... Go find a tab of the song, two or three if possible. Check the tuning-the song you are trying to do may be in drop D or other alternate tuning. If that doesn' t solve it, peek at just the first chord/note. Use that as a reference point to hunt out the rest of the notes. If still stuck, keep peeking. So what if the first few songs you only pick out one note on your own?
When I'm trying to learn, I get the reference point i.e. first note. Then, I know that most songs are either in major or minor-so I stick my 2nd (bird) finger on the root and play the major scale over the song. If those notes work, I''ve got most of the notes to play off of. Leave your hand in position and keep pecking out the notes. If it's just not working, try the minor position (first finger on root) and play around on those notes. First thing my teacher did was teach me hand position, one finger per fret. If you don't do this now would be a good time to learn. This has helped me pick out 1/3 of songs.... if it's blues songs, or rock, knowing 1/4/5/4 progressions helps too-after you've practiced them 1 million times you'll start recognizing them by ear.
Another thing to check is that the recording you are trying to learn off of is in tune. I was ready to throw myself against a wall because I *couldn't* find ANY of the notes for one song. Usually I can just keep playing up the neck and find at least one! But nothing I did was right...... had a guitarist listen and guess what? My recording was almost perfectly a 1/4 step out of tune. No way could I learn off of that!! So it was off to find an in tune version for me

. Now, what you can also do in that situation is tune your bass to whatever that recording is. If the recording is 10% flat, figure out the first note and tune off of that. Once you learn song, put bass back in tune and viola!
Learning by ear takes experience and practice......I'm not there yet.

