I found it liberating, actually. Playing fretless sort of points out technique issues with an exclamation mark, so i'm more inspired to work on technique. It also makes me listen much better, and therefore connect with both music and instrument more.
Challenges: teach yourself to play with your ears exclusively. One of my warmups is to take any note, let's say a C, and play every C on the fingerboard in random order, without looking. Really helps with developing positional awareness.
Your fretting hand may need adjustment so that you use the tip of each finger exclusively. You'll find that the tone bevomes vague and muffled, drastically so, if you lay your finger more flat on the fingerboard.
And nice thud or attack (that little click) i find much more difficult to obtain with finger style playing on a fretless, especially with roundwounds. The notes tend to envelop rather than 'click', easily giving the impression that you're either dragging the groove down, or overcompensate and consequently sound way too pushy for typical bass requirements, tonewise. So precise phrasing is key, and some experimenting. I find a very light touch, which is hard for me to do, to work the best.
But all in all, nothing satisfies like playing fretless. You'll have the ability to breathe life in each and every note. I find the freedom to go anywhere, and amount of expression at your fingertips, so rewarding, inspiring and musically liberating. Good luck!