Yep. Tell the keys player to lay off the left hand. Better yet, tie it behind his back.
I back up a choir and we have 3 piano players (not at the same time but in the rotation). Two of them cannot play the right hand without playing the left hand. That's all they've done, and hard for them to overcome this.
So in the end, the piano player and the two acoustic guitar players overplay like they are the band themselves. The guitar lead noodles, basically solos for the entire song or hymm, and has a tendency to head into inappropriate scale and notes at times. This makes it sound like we're playing in the wrong key at times.
So in the end, nobody sits in the mix well at all. The music director doesn't see this issue that much, and some have the view of "the more, the merrier".
But hey, we've fired the drummer! She rarely shows up anymore.