My experience has taught me that if you turn down too many of the gigs you don't really want to do, then the ones that you do want to do can dry up really fast.
My Dad told me years ago to never turn down a gig if I was free to play it, and I think he was right. It's those crappy gigs that get you a reputation for being willing and able to work playing music, and you never know who'll be watching, who you'll meet and what the results of that might be in the future. I'm talking in general terms now rather than this specific situation, but this still applies in this case, I think.
And Bill slides in for the finisher. Much wisdom in this post.