I have done this a time or two... with somewhat qualified success. If the baffle board is plywood, it's next to impossible to do it with sanding. I used a rotary rasp, about 3 or 4 inches in diameter, chucked into a variable speed drill. I got it done eventually, but in retrospect I decided it was more work than it was worth. If I had to do it again, I'd probably try to figure out a way to use the same template-and-router technique that I use to make speaker cut-outs in new work. I'm imagining that sanding or using a rotary rasp would be a more reasonable approach if the baffle board is MDF.
In another thread, a poster mentioned that he had had success just rounding over or chamfering the front edge of the opening all the way around. I haven't tried that yet, but my memory is that might well have worked in my case as well. In his instance, as well as my previous encounters, we were trying to enlarge the cut-outs to accommodate EV drivers.
Somewhere on-line (sorry, I don't know where) I ran across a speaker parts supplier that sold some "speaker adaptor ring" thingies that might offer an easy solution if you can handle the extra 3/4" it adds to the height of the speaker above the baffle board.
Good luck with your project.
Edit: I see BasturdBlaster weighed in with the same solution of just rounding off the edge. I bet that might work.