A hurdle that players of fretted instruments struggle with after a certain point is that they learn music as a series of repeating physical patterns. This really accelerates facility on their instrument, but can be a limitation when you step away (as you describe). Most people learn to play guitar chords by shape and not how to build chords with intervals, based on the chord qualities. If you ask a reasonably good self taught guitar player to play a C7 chord, they can. If you ask them what notes make up a C7 chord, they have to go to their guitar, finger the chord shape and then count up from the open strings to identify the notes.
I suspect this may be some part of your problem. "up a 6th or down and minor 3rd", is plenty descriptive for a lot of people.
What I suggest is start looking at what notes make up the scales in these keys. Key of C - C Major chord, no sharps or flats, all the naturals starting on C. All the white keys on a piano. If you played a minor scale using only those same notes, what would it be? It would be A minor. All the white keys on the piano, starting with A are the same notes as C major. That's WHY it is it's relative minor.
The key of F has one flat (Bb), so does the D natural minor scale.
The key of B Flat has 2 flats, one at Bb and one at Eb, so does the G natural minor scale.
Maybe working through how these scales are built and how they relate to one another will give you a different anchor. Maybe it's best to work through this away from your instrument to do it without the crutch.
Just another set of ideas.
For me, all the "modes" were just too much to memorize. Once it made sense to me musically, it was way easier for me than all of that.