Musicians and audiences found it delightful and irresistible to play/hear syncopated rhythms, to play/hear those "blue notes", and to lose inhibitions about the "proper" way of doing things, glorying in improvisation and passion-in-the-moment. A lot of people, too, got a kick out of the idea that this sensuously thrilling music they were enjoying was a sort of flip of the finger to "serious music". It got called "jazz"........ It came out of the streets and crossroads and whorehouses, not the academies. And it came out of a relationship between player and listener......So the connection between jazz and the songbook is historical. Those were the songs of the day when jazz was at its popular peak. They were the material at hand. And, as it turns out, the 32-bar form with that particular type of ii-V-I harmony and those sorts of melodies turns out to have been a pretty fine structure for blowing. I think they're gorgeous. They're before my time but I've been hearing them my whole life. I think in the future there will always be people who find the music of that period, the American popular music from the 20's through the 50's, the jazz and songbook combo, beautiful and compelling.
But time moved on. History happens. Jazz audiences are small, .......when you talk to listeners these days, the listeners who don't like jazz (i.e., most listeners), the number one complaint they've got is that it all sounds the same, you can't tell what tune is being played. If that's true, why would it matter whether the tenor player is breaking a sweat on "Airegin" or "Smells Like Teen Spirit"? The listener is alienated from the music either way.
Given the way history happened, I think jazz is exactly where it should be right now. One stream pursues the repertory aspect, preserving and vivifying The Tradition every day, and passing it on..... But the socio-cultural realities that bound player and listener to the jazz way of doing things, the realities that gave us Duke and Basie in the heyday -- they are dead and gone.