Barry says, "Well we're just hitting it off like a storm, aren't we Clive?" And then "As a matter of fact, I think we'll just leave."
I was watching Robin through the interview, and he looked somber and unhappy from very early on, while Maurice and Barry are still smiling and joking along.
My read on the interview is this: they were ready to joke and be self-deprecating and play along, but Robin was absolutely not in the mood and I suspect Barry could feel the glare coming from his brother. Maurice was increasingly sidelined in the conversation, so there's Barry trying to keep it up by himself and it just goes on and on with the host treating them like a joke and finally he just doesn't have the energy for it anymore.
I think they wanted to talk about exciting new things they're doing and after nine minutes they still didn't have the chance to do it, time's running out and the prodding doesn't stop, so he's not going to keep it up indefinitely. I agree that the "tosser" line got to him also; it's one thing to say "I was a tosser back then" and quite another for someone you don't know to say, "Yeah, you were."
The thing about a band like the Bee Gees at that time was that, yes, they were very often the butt of a lot of jokes, and they had come down a lot from being ultracelebrity hitmakers. When you're in that position, you don't want nine minutes of having your nose rubbed in it. You can joke about it for a couple of minutes, then you want to turn the page and say, "So that was then, here's what we're doing now," and the interview just wasn't going there. I don't think they would have taken the interview if they expected just to have their noses rubbed in it.
If it was, Maurice wasn't in on it. Robin was ready to leave pronto as soon as Barry got up. Maurice was awkwardly fumbling around taking his mic off. With the other two already offstage, Clive is sort of pleading with him to stay and finish the time slot, and Maurice says, "I don't do impressions," which I take to mean that he wasn't going to pretend to be Barry, who had been getting all the questions aimed at him. So he probably wasn't happy either.