Nah...done it many times. Can't tell. I've put 'em in, and I've taken 'em out, no difference. If it's bolted on tightly, you're fine. The surface area is meaningless for tone/sustain. Go ahead, disagree, I don't care. I've done it enough times to know.
I do agree that a tapered shim is best. Helps prevent movement by increasing friction, and also discourages 'Fenderitis,' or rising heel we've all come to know and love(at least us Fender guys).
Another, lesser known reason to adjust neck angle is to correct an intonation issue, where you have run out of bridge adjustment(probably due to incorrect initial assembly). You can gain back a surprising amount of intonation adjustment at the bridge this way.
Another reason is some folks prefer more, or less, distance between the strings and body. You can tilt the neck for this, and then readjust the bridge to wind up with the same fretting action as before the tilt.