Really, the issue at hand isn't so much 'downsizing' as it is 'downgrading.' I don't mean that in regards to Fenders vs. Spectors as I think each has it's own unique merits, but in terms of the flack you're occasionally getting from audience members, or, more to the point, disgruntled other bassists who are unable to appreciate your playing because they're so stuck on what you're playing with and through, the tacet assumption here is that it's not so much an issue of a smaller rig (which could be just as, or even more high end than the rig you're alread using) as it is a less expensive rig.
Which is garbage.
While there's something to be said for being able to work with whatever's lying around, which comes in handy if you live in an area where clubs provide drums and bass rigs and you have to find the closest approximation of your sound in the fifteen minutes you get between bands, when you have the chance to choose your own equipment there's no reason why you shouldn't use whatever you like. I've read more than a few interviews with pro players who advise to get the best equipment you can afford. And 'best' may not mean 'most expensive' 'Best' is a balance between tone, reliability, portability and suitability to the gig, so whatever works for you without breaking the bank is valid as far as I'm concerned. Given the number of potential barriers in any given venue to the clear representation of your tone as you want to hear it, every advantage you can give yourself (again, within the bounds of your budget) is justified in my opinion.
It should also be pointed out that we're talking strictly about live reinforcement here. In the studio you never know what's going to work. You may find that the Neumann in front of a 410XLT gives you the exact sound you want, or you may find that on one tune you blow the dust off that Polytone Mini-Brute and throw an SM57 in front of it, just because the sound works for that particular tune. In the studio it's a controlled acoustic environment so you don't have to worry about things like 'am I loud enough' or 'is the sound man EQing me with his idea of what I should sound like' so the concerns are different.
So yeah, use whatever you like. I used to go through the same thing in my last band where I ran a smaller but similar setup to yours with a WT-300 as a preamp into a QSC RMX 1450 powering a 210XLT and a 212XLT, both 4ohms. But really, the only time I felt kinda dumb was when we'd be traveling to an unfamiliar venue and we'd show up and the stage would be the size of a dining room table and the venue would be so tiny I'd have to stoop to get in the door (the Cave in Chapel Hill comes to mind...) So fine...maybe leave the 210 in the car. Or screw it and just turn down.
But as far as people getting on your case for having good gear, unless you're trying to fit a specific image I wouldn't pay them any attention. If they're going to spend more time analyzing your gear than listening to your music as a whole then changing your gear isn't going to help, it'll only change the thread of discussion. Concern yourself more with the people who don't know or don't care about the gear but are there to listen to your music.
J.
Heck, I'll trade ya my Peavey 115 and Hartke 210TP for your 2 410xlts. No one will complaign about your high-priced rig again. 
