Maybe. Only you can answer that of course. But I will say, when I had just my fretless and was waiting for my fretted build, I really enjoyed playing fretless and didn't use the fretted very much when I first got it.
Sometimes I think I'm meant to be a full time (or majority time) fretless player.
I can't afford to have bass just for goofing off. It would need to be an instrument I would actually use at church. I could get the hang of fretless if I put in the work, I just have to be honest with myself, and wonder if I will really put in the work?