size does matter, and you have to spend time with different speaker cabinets with different configurations and cabinet designs to decide what works. example: Ampeg's famous 8x10 is a sealed design that doesn't bother with a HF unit. on the other hand, Ampeg's own PR410HLF is a triple-ported 4x10. there are legions of players who will swear by either of these cabinets and yet they sound pretty different from each other despite both using 10" drivers. you cannot simply say "well i tried some random 4x10 at a store and i decided it sucks."
Personally speaking i have a preference for larger speakers and unconventional cabinets. my proposed future rig's cabinets will be a large W-style folded horn, like an old Orange, Fender 400 PS bin, Acoustic 301 or EV Eliminator and a Traynor 9x8 beside it. i'm lovin' it.