A couple of thoughts;
One is that, at least the way my ethics and my sense of myself work, we are one far more certain ground when we accept that we are all at least potential offenders, in one way or another, if not in this area than in some other. Not just men but women as well, of course. The person who is in the most moral danger is the one who presumes himself to be in the right and immune to wrongdoing. There has been a tendency to denigrate the phrase "there but for the grace of God go I," but understood rightly, what it means is, "That person we're condemning could very easily have been me." The "not all men" reaction insists on the opposite; "I am nothing like that offensive person, don't blame me."
The other thing is that "not all men" and similar responses tend to segregate the offender off from the rest of us as something "other." We often turn to the language of mental illness to brand them - "that guy was a sociopath, a psychopath, insane." When we do that, in effect we deny their wrongdoing a rationale or motivation. There's no sense or logic to what that person did, it was just madness. I'm not mad, ergo it has nothing to do with me, and there's nothing I should be expected to do to change or prevent it. Some people are just psycho.
But the reality is that wrongdoing DOES have a rationale and a logic to it. Within a certain framework of values and assumptions about sex, gender, power, pleasure, status and so forth, harassing or assaulting a woman (or any number of other kinds of bad behavior) make PERFECT sense. The behavior isn't an inexplicable, irrational manifestation of mysterious mental illness, it's a strategy for what looks like "success" if you internalize certain values and assumptions and reject others. That framework comes from the culture. And we are ALL contributors to the culture. That means we all do bear some responsibility for whether we contribute to a culture that encourages or discourages the decision of "some men" to behave the ways that they do. That's a conversation we have to be having.
A few months before all the recent scandals broke, my drummer came back from a tour supporting a young female singer. It had turned into a nightmare. The manager/producer (and, if I recall right, bass player, I'm sorry to say) in the band turned out to have booked the tour largely as an opportunity to try and get into her pants, though she was half his age. I won't go into details but there were a number of things he tried to manipulate with the rooming situation and off-time to pressure her into being with him. Fortunately the other band members pushed back and supported her and finally the whole tour came to a crashing end with half the dates canceled and no one talking to each other.
My drummer came back and told us this story, and everyone was horrified, but honestly I think my horror at it was a little different from his. His reaction with disgust that anyone could even think this way. To me, though - also being a man twice this girl's age - I could completely see what he was thinking. It made absolute sense - a young, pretty, vulnerable girl, depending on you to make her dreams come true? That's not hard to see the paths his mind went down. To me the horror was not that he could think it in the first place, but that he wouldn't have known better and, by his age, had the maturity to set up the safeguards to prevent his behavior from following his fantasies.