It would help to see your proposed schematic (not a wiring diagram). The grounds aren't the problem - the real problem is to remove the loading of the pots when you want to go straight to the jack. I have a feeling (but need to see a schematic), that you will actually need more than 4 poles to do this. Maybe not though.
I recently drew up a schematic (maybe for you?

) for a 3 way P, P to jack, and J+P. This took 3 poles.
Edit: It was you -
3-Way Wiring Question. Well, I may not want to take the time to draw anything up after the lack of responses in the previous thread.