Definitely a good point. And the electronic adjustment (e.g., some kind of EQ) will add its own group delay, further clouding matters.True, there are other things that make it even more difficult to hear.
E.g. if you electronically adjust the response of a closed cab to mimic that of a ported one(exact same driver) you may get the same frequency response but still it sounds very different. The driver in the closed cab will have a larger cone-excursion in the freq range where normally the port would aid the driver, this will cause added harmonic and intermodulation distortion. The ported cab on the other hand will have lower distortion but worse transient response in the freq range where the port aids the driver. There's no free lunch here, each enclosure type has it's own cons 'n pros.
Distortion is definitely a difference.
And this raises an interesting issue. If you're using an amp with built-in voicing, such as "scooped mids" typical of many bass amps, then the amp itself is affecting your transient response.