Could try using higher grit paper, or steel wool. I would avoid going straight to a super fine grit though, as you'll still have deeper scratches/swirls under the smooth bits. My impatience has caused this to happen...
Way to avoid is going up grits in stages, but this causes another issue; you could potentially go through the finish, if you sand too much.
Is it natural finish, or block colour?
One more thing, the higher grit you go, say towards 0000 steel wool and them polishing rubber things, you may get a shinier finish, which would be counterproductive
Good luck, sir.
All the best.
PS, I'm no expert, so I would wait for the cavalry if I were you