Every piece of gear in the chain, from the cables to the speakers, affects the phase of the signal. Speakers affect phase differently at every frequency. This chart shows the phase response of a 1x12: You cant hear the shifting phase because theres only one source, and phase effects are only audible when there are two sources with different phase. As for a second source, heres a 1x15: You can see that the phase response is different from the 1x12. Thats not just because the driver sizes are different; you get a different phase response even using the same driver in different box sizes. Now look at the 1x15 phase response overlaid on the 1x12. If the two charts were the same there would be no problem, but they arent, and thats wherein the problem lies. Their combined outputs will augment each other at some frequencies, detract from each other at others. Thats why well engineered speaker systems dont run different drivers in the same bandwidth. They run different drivers in different bandwidths, using crossovers to be sure that multiple drivers always complement each other. The above charts show phase response of sealed cabs, as its easier to see. This chart shows phase response of a ported cab, which is more illustrative of how complicated the interactions between different drivers and different cabs really are.