I say go for it! but let me share a bit of my experience with an older musician situation. I was telling this story to a friend just the other day... When I was about 23yrs. old, a buddy of mine (he was about my age, maybe a year or two older) had this sort of funk/rock/fusion (but with some emphasis on odd time signatures... yeah it was kinda weird) band that he was building. He had been jamming with the drummer for a few years before I stepped in, the drummer was 43yrs. old. I had no problem at all with that.
While at practice, we were figuring out a part and the drummer said something like "it's 11/7 time; 11 beats 7 times" and I responded "No, there's no such thing. Top number is how many notes per measure, the bottom number is which note gets 1 beat, so the bottom number can only be a 1, 2, 4, 8, 16, 32 cuz of wholenotes, halfnotes, quarter notes, eighth notes, sixteenth notes, etc..." (though 16, 32 are rarely used and yes it can be 1 i guess cuz of wholenotes, but that's why its called "music theory", theoretically true doesn't necessarily make it fact/practical). His response was "no dude, I'm a drummer I've been playing longer than you've been alive. I know what I'm talking about." The guitarist couldn't read music so he couldn't really say. I was in marching/jazz/concert band in highschool, I even won an award for Excellence in Music Theory (I didn't mention this to him in that conversation, he was obviously pulling the "i'm older wiser" card, so he wasn't gonna back down and I didnt' feel like getting into a music theory lesson with him). There were several other reasons for it, but I left the group shortly after that; I got zero tolerance for that ego bs, and he made the age gap an issue, not I. It was probably more ego than age, but like I said he was justifying it by saying "i've been playing longer than you've been alive".
People pull that crap regardless of age. I still say go for it and jam with those older dudes. I just thought I'd share a situation where age became the issue eventhough I, and seems like most people in this thread agree, don't think it should be. rock on.