Let's do some math!
I'm going to use a 4-string with 34" scale and D'Addario XL's as an example since they make their tension readily available for each pitch.
I'm going to use the 50-70-85-105 set for an example since anything heavier would put some serious tension on the high strings.
A 34" scale bass tuned standard EADG with these strings at the specified gauges would have ~202 lbs. of total tension from the strings. Note the D string alone would be almost 60 lbs. tension - yikes.
At drop C CGCF tuning, the same strings would have ~153 lbs. of tension. So basically about a 25% reduction in tension on the neck. I'd say that's significant.
One thing OP didn't mention is how often is it being changed back and forth?
If it was just for a few songs, I can't see it being a big deal. But I would also find this cumbersome especially in a live setting.
If you're talking about having it in EADG for a few days, then CGCF for a few days, etc., I could see that making the neck move around if it's not a strong neck.
Even if this didn't cause any physical problems with the bass, I'd personally stay away from it just because you'll never have optimal tension and playability for both tunings with the same set of strings. I realize not everyone has the luxury of owning basses for each tuning they want to use, but I was able to get myself a bass setup specifically for CGCF and I love having it feel the way my standard tuning basses feel.