Actually, it used to be that phantom power could damage mixer outputs a long time ago and it's always a good idea to check with the manufacturer. However, most, if not all, mixers have designed their outputs to withstand phantom power applied to them in situations such as yours. I did check on the X32 and yes, phantom power will not harm the outputs. I've read about "DC blocking caps used to protect the output circuitry". I don't understand the design parameters, but early on in my pro sound career, it was a concern.
We owned a Soundcraft Series Five mixing console, designed about 20 years ago (and a wonderful-sounding analog console),
that was susceptible to damage to the outputs from phantom power being applied. We started seeing video trucks with cheap little Mackie mixers with a global phantom power switch. Way back then, I did smoke an output from such a situation. I figured it out when that output's meter (a group bus output going to the video truck) was acting very strange and the truck reported really bad distortion on they end. I checked their line and sure enough, it had phantom power. I pulled out a Sescom IL9 (a 1:1 isolation transformer for that line and used another group output. All was well. $125 to repair that group output, though. I made a standing rule with all the audio guys, ALWAYS, ALWAYS check any line a video truck (or anyone else) hands us.
If it were me...I'd get and carry a 1:1 transformer (from Rapco, Conquest, Whirlwind, etc) and use it (I do). Don't bother me with the "over-cautious, unnecessary, waste of time, etc" comments. I don't disagree that it probably is, but it can't hurt. However, I'm sure you'll be fine without it on the X32.
EDIT: I just noticed the post about the combo input jacks. On my little IEM mixer, instead of a transformer, I use a simple TRS-to-XLR adapter and use the 1/4" TRS side of the input (2 channels have XLR and TRS line inputs that no phantom power going out on. You can do the same thing with the TRS side side of the input jack and accomplish the same thing.