First you say "I just don't always use my ears" and then you say "also I spent about 13 of the last 15 years listening to mostly free jazz". Like it's two different things. If you are not playing what you are hearing, you are not playing music. If you are not listening to the people you are playing with (generally) you don't have a chance of creating a musical moment.
There are a couple of threads going you might want to check out, Mike Da Mook's thread on soloing and walking and the thread on memorizong standards (although I don't really agree with a lot that's been said in that thread)
Believe me, I've been EXACTLY where you are. Your playing has gotten to a point where you sound like you can play and people call you for gigs, but you can't seem to make the conceptual leap to get where people you hear on records are playing. There are some things that have been said here that I have NOT found to work, in my own personal experience.
My aim is to build my vocabulary - it's not about vocabulary. That was my big revelation after coming north. I would paly duo or trio with all of these players who had great ideas, tune after tune, night after night. And then I would be hard pressed to come up with a chorus or two that didn't sound like gibberish. So I thought " Must learn more vocabulary". So I studied with a great player, we learned all sorts of scales, modes, chordscales for altered chords, modes of the melodic minor scale, blah, blah, blah. What it gave me was more notes to choose from but no real clear way to determine what notes I really wanted to play. It's not about vocabulary, it's about meaning and intent. You can pick all the notes that are supposed to work over a dominant chord you want to, if you are not clearly hearing what you want to play it is IMPOSSIBLE to come up with a coherent and intelligible line (bearing in mind the monkeys/typewriter/Shakespeare scenario. But do you really want to wait for happenstance or do you want to be self directed?). If all you can clearly hear is a limited vocabulary, but you REALLY hear how the chords move and REALLY hear the notes in your limited vocabulary within the harmonic progression and have the wherewithal to play those notes you clearly hear and know (can identify with clarity) on your instrument, the suddenly what you can play communicates to the other musicians and to the audience WHAT YOU ARE HEARING when you are playing say OUT OF NOWHERE. You're not stumbling blindly through the woods, you are exploring a path that is taking you towards the destination (whose location you know). It may not be the most complex path at the beginning, but it will be evident for anyone who is watching (AKA listening) "Oh, he's going there and that's how he's getting there".
its no good having ideas if you don't have the ability to reproduce them. - learning technique is not going to give you ideas. If you hear something clearly enough, you are going to do WHATEVER it takes to get that idea out, even if it means playing it on one string. Charlie Haden is a great example, his technique is fairly atrocious. But he gets the ideas out. Sure, if you come up against your technique on a gig (you don't have the chops to play the idea at tempo, your fingering was a mess cause you haven't spent enough time on two octave harmonic minor scales with string crossing descending etc.) you shed what gave you a problem until it is no longer a problem. But the idea came first. Because you heard something clearly that you wanted to play.
So my ears are perhaps over-extended. A ton of subs or complicated structures sort of takes away from my ability to be a well-rounded musician. I just never thought playing 1 5 1 was very cool, and missed out on a great deal of subtlety because of that- my first jazz album was the shape of jazz to come by ornette.
I'm not sure what you're really saying here. If you are really really hearing subs or "complicated structures" you will use them in the way you are hearing them. If all you are doing is dropping subs in because you think you are supposed to or because somebody told you it was hip, it sounds like you're just not listening. The only reason you should play a note is because that is the note you are hearing in the context of what you are doing.
Think of it like a conversation. Not just a "hey how ya doing how's the weather what did you see on TV last night" but a conversation with somebody that is involving and deep and you really want to hear what they have to say cause it's just interesting as all get out and they want to hear what you have to say because you have some points that they really haven't thought about in this context but now that you mention it that really is true. A true and honest exchange of thought and feeling. When you're in that kind of conversation, what you bring to it is the things that you have thought about a lot, that you express in the EXACT and SPECIFIC words that express that thought in your own true voice.
So you can't bring a lot of words that you don't really use in your everday conversation, you can't drop in phrases you read in a book recently or something that somebody else said. Yuo play a sub because that's where you heard the tune going, either in something the soloist did or the accompanist did.
Anyway, I gotta run and meet my girlfriend, I'll try to finish up in a couple of days or something.
But this is exactly where I was a few years back, and the thing that got me on the other side of that brick wall wasn't learning more vocabulary.
later.