You're not too picky. You should be able to get a nice low action without buzzing. My old Stingray does this without a single issue. In fact, I remember the first time I got a USA SUB, I was very impressed by EBMM quality of necks and fretting (it was my first EBMM). I've never been able to get such a nice consistent action before it, and my next Stingrays did it just as great. I don't know how the new ones compare to the old ones in that regard, I haven't tried a Special.
It might settle down but if it doesn't, ultimately it'll bother you and you will want to move on to the next bass (I know I wouldn't settle for that). Are you able to replace it?
Honestly, I like the bass so much otherwise that I am willing to put up with some buzz as long as it doesn't get too bad. I am just worried that my neck might be a lemon, which would be a real drag. I am not sure what I would do at that point.