If you have a single pole at 40Hz, the natural frequency is 251 radians which is a time response (tal) of 4ms (not 2ms, must have mis hit the calculator the first time). This is just a ball park estimate. Later in the post I state that the phase lag could be even greater depending on how the filter is designed.
A 4ms tal, means that if you hit it with a square wave, you will get a response that is 63% of the way there at 4ms, 86% of the way there at 8ms, and 95% of the way there at 12ms. Again this is an order of magnitude calculation.
Also remember music is not sine waves, shifting a sine wave 360 degrees is nothing. But phase shifting a time varying input by the same amount is a delay that may be audible.