It isn't really ownership. It's more about whom the drummer drums for, whom he or she has the primary relationship with. Here's what I think the difference is. For some of us, saying "my drummer" means the drummer drums for me personally, and "our drummer" means the drummer drums for us (ie, the band). In a band, for the most part, the second usage seems more appropriate and respectful than the first. If I'm in a band with other musicians, I'm not the guitarist's personal bassist, or the singer's, or the drummer's, or the keyboardist's, or the sax player's; I'm the band's bassist.
But again, I do think this is pretty minor on the great scale of things. I personally do find the "my" usage mildly annoying, but I, like most people, have many more important annoyances just about every day. I only post about this because isolation is boring.