Usually it's not the swearing that I find offensive but the attitudes of the comedians. Frankie Boyle is a good example as some of his stuff is brilliantly funny while some of it is just trying too hard to be offensive.
Most of the comics I like are fairly clean or at least I don't notice...