well... bear with me. The following statement is an assumption.
I'm assuming that the nut is in two sections, the piece that falls off is like a bridge saddle, minus the screw holding it there when there's no string tension.
Then the two screws (or allen key screws) on the Just a Nut adjust how high it sits from the base of the nut slot. j
If my guess/assumption is correct, then yes, your nut part would fall off if you took all of your strings off your bass.
Maybe you could take pics of your specific instrument and take some with the strings on and then take the tension off so we can see what actually comes off the nut?
I'm curious about your dilemma and also hoping my assumption is correct, and if not... I'd like to find a way to rectify it if something else is going on entirely.