I don't think there are any drawbacks to that structure of not having too much structure... you just have to call people who are capable of listening and reacting to each other in such a way that music is made.
I have a group of close musician friends that I have much experience working with and know what they each bring to the table, so I dip into that pool when I need to put a gig together like that. Some of them have done lots of stuff with me like Tim Miller having played on all my solo albums to date, and Justin Vasquez who I work closely with not just in my own band, but with his music, and various outside project such as:
The Live Archive Podcast | Playing and discussing rare live recordings from the most extensive archive in existence
I've been slowly stripping away information from my songs to bring them down the bear framework of a piece of music, and try to allow the people I hire to play that music inject their personalities into the performances. Sometimes it's very successful and sometimes not. Either way we all learn a lot from each other and continually grow as human beings.
I happen to love Cliff Almonds playing and think he has something very original to bring to the table. We're both so busy with other projects that we rarely get a chances to play together, so it was a perfect opportunity to check in socially and musically with an old friend and make some music. And yes, everyone flew in for that one gig which was quite epic. It meant no rehearsal, and it also meant choosing music that people had already played before, or that people could learn just by listening to some MP3's that I sent out before hand.
I hope that sheds some light on the situation.
Thanks,
Janekl