I have always felt the same way. Unless it was with, say, a close friend where there were promises made.I personally never got seeking harm for the one who isn't one's spouse. It's illogical. That third person never made any promises to you to not do that. Not that I wouldn't be mad, but it would be misdirected to the wrong person if I found/learned about my wife with some guy. He should know better, but she was the one who promised to know and do better.
But even there, the deepest promises were the spouse’s.