It seems to be standard operating procedure to build these with butt joints and add the baffle last but to me it seems wrong. I'm inclined to use dado joints, build the box around the baffle, and drop in the back panel last. It's more work, requires a router and a carefully clamped straight edge but it seems better to me. One issue with this approach is that the plywood is actually 15/32" (or the metric approximation?) and a 1/2" router bit will leave a gap ( hopefully consistent to one side of the board) that needs filling.