Update
Today was a long day. I first visited one of my two favorite luthiers. He said that everything is fine, he checked the electronics, he tried it with the two bridges, he checked the frets, the truss rod, etc. He thinks it's maybe the light alder body, he's also saying that it's maybe a neck resonance problem. He also tried Bruce's trick regarding neck stiffness, there was barely a noticeable difference in the relief with the bass detuned by two steps.
Then I went to the luthier who built me that bass. He also checked the electronics, the pickup, the truss rod, the frets, etc... He says everything is nice. He sanded the neck heel and neck pocket. With the heel finish removed and the neck pocket clean, it didn't changed a thing. He then decided to remove some material on the head stock, the head stock is a little bit thinner now and it helps but it also changed the situation. We spent some time adjusting the pickup height and now the volume is identical on the E and A string. Here how I would describe the problem now. I hear it like the roundness of the E string is not the same as the other 3 strings. It sounds like the E string as more definition or clarity than the other 3 except. There is no more difference between the first 3 frets and all the others on the E string. They are now similar, more defined, clearer, no as round as the other 3 strings.
Next step, the luthier thinks is to shave some material off the back of the neck which is like I mentioned earlier, thick and stick.