The sound/tone/delivery of singing can evoke a completely involuntary (and utterly honest) aesthetic response from a listener. Doesn't mean the listener has to feel that the band or songs are bad - just that the main point/focus of engagement for most 1st listenings to any music is whichever instrument, primarily, carries the melody or the lyrics (commonly both, of course).
Sometimes it can mean that the listener's response is that they think the music is dreadful, for sure, but it doesn't automatically equate to that. Aesthetics is a peculiar area of art - it's often viewed as being very binary but a large part of post-grad study in teaching music (here in the UK) focussed on strategies to make something students initially recoiled from appealing to them. So, not liking someone's voice/delivery is a really honest response and tho peer-pressure can sometimes prompt a listener to try again (sometimes growing to like the voice) there's no reason why anyone HAS TO LIKE ANY SINGER/VOICE.
I can't listen to Zeppelin because of the singing, love a lot of the music but can't stand Plant's singing on their songs - but I loved the stuff he did with Alison Krauss. I know, sacrilege right!? Geddy I can dig but can easily see why someone else would respond to his singing the way I do to Plant.
I am of a mind it's to a singer's credit to evoke such a response, positive or negative - it proves they've def got their own thing going on!!