I could be wrong, but if you do that doesn't that double the impedance make the amp not as loud?
No. Unless the cabs daisy-chain in series, which would be very unusual. Generally, it is a parallel connection, which works the same as connecting two cabs with separate output jacks.