Certainly.
The better (IMO) way to state it would be: Regardless of driver size, different drivers will sound different in the same enclosure much like identical drivers will sound different in different enclosures.
I chose brevity. Besides, mine was a question, not a statement.
Whether the emphasis is put on the driver or the enclosure makes no difference to me. I simply take exception to the thought process that goes "Driver diameter 'X' yields tone 'Y'."
So do I. I also take exception to the thought procees that any driver will yield tone "Y". In the real world. Doing it on paper's easy.