Sorry to hear about your frustrations. I have to say, synth pedals are difficult for pedal builders to get right. Like a lot of other guys on here, I've tried quite a few and none are perfect.
There are certainly lots of players out there, including some session guys, that are using the SYB-5. Have you compared your results to others' on Youtube? There are some videos where you can see the knob settings while they are playing.
With your SYB-5, if you are getting no synth sound quite a lot, then it might be that your bass's volume is not loud enough to trigger the synth. How much have you turned up the two pickup knobs on your bass? If not much, I would try turning them up fully, then turning the volume/gain on your bass amp down.
As I mentioned before, sometimes synth pedal play no note or the wrong note. The SYB-5 can let you mix in your normal bass sound, so that if the synth freaks out, your bass sound is still there to carry the bass line. You can do that by turning up the Direct control (outside part of the first knob).
According to the manual, modes 1-9 are synth modes, where an oscillator produces the synth effect. Try modes 10 & 11 instead which are waveshaper modes - they change the sound of your bass instead of creating an entirely new oscillator sound, so they might track better.
There are lots of alternative synth pedals out there. A fun one is the ElectroHarmonix Bass Micro Synth, which has lots of sliders to play with. I have this and it's great for 70s Bootsy/Moog type synth sounds.
I also have a Markbass Super Synth which is digital and can store different presets - it's good and sounds fat, but is difficult to edit on a gig. If you are just working at home or in a studio, then you can use a computer & USB cable to edit the sounds. I'm now looking for an Akai Deep Impact - discontinued but still popular and goes for $600 on Ebay!
Don't be disheartened about your SYB-5. Even if you decide to sell it, you'll find the second hand market is good for pedals in good condition (with box, manual etc). One reason why this forum is so active is that we can buy and sell lots of pedals and find ones that work for us, without breaking the bank. Good luck!