Left Hand Difficulty / pain - in first pos

I've got normal sized hands. Whenever I have to go lower than the 5th fret, it's incredibly tough and my left hand starts hurting pretty soon.

As you all know, there are a number of progressions/songs/licks that just lend themselves to playing in first position. Being there minimizes same finger string skipping.

Will playing more in that spot help, or should I just anchor at the 5th fret and figure workarounds?

 

Centered on my solar plexis / Jazz position.