Impedence, power handling, sharing the same box with different drivers, etc... It's not likely to sound great in the first place, and the chances of damaging one pair or the other or both is relatively high (and then once one pair goes, you're more likely to blow the other pair). I mean, if you play it at bedroom volumes it's probably not a big deal, but I wouldn't try to push it hard, and if you can't get the most out of it, what's the point?