It's mostly about amp settings, mic placement, and mic choice. Remember that standing in front of an amp sounds very different compared to putting your ear against the speaker. So the mic and the player are hearing two different things. More often than not it's necessary to adjust the amp to favor what the mic picks up as opposed to what the player hears. If the mic'd sound is good you can always give some guitar back to the player through the monitors.
But before you touch the amp let's look more closely at the mic. The SM57 is a perfectly suitable mic for guitar as long as you know it's two main quirks:
First and foremost, like most dynamic mics of its kind, the 57 has a pretty sizable "presence peak" around 5kHz (more than 5dB) which is responsible for most unpleasantness related to guitar. Address this first before grabbing the highs (12kHz shelving filter) prematurely. Also keep in mind that the closer you put the mic to the center of the guitar speaker, the more this frequency range will be exaggerated. Remember the very top end of a guitar speaker's fundamental frequency response is about 5kHz (harmonics notwithstanding).
Second, 57's have a fairly sharp drop in low end starting at 200Hz so by the time you reach 100Hz you're -5dB and by the time you reach 50Hz you're -10dB. But before you go crazy with the bass knob, this isn't as bad as you think. The lowest fundamental frequency on a standard tuned guitar is 80Hz so grabbing the bass knob on your console (typically an 80Hz shelving filter) will do very little to fatten up the overall sound but is fantastic for making mud. All that's needed IMO is flatten out the 100-200Hz range by boosting a little 150Hz. Be careful here because too much bottom may sound good by itself but doesn't always work in a mix.
If you take care of those two quirks, the 57 is remarkably flat between 200Hz and 4kHz which, coincidentally is where most of the guitar sound lives. Meaning if the sound source is good and the placement is right, the 57 won't let you down.
If you still struggle with a thin guitar sound regardless of mic placement, then a different mic can help. I like the Sennheiser 421 on guitar as well as the Heil PR30 and Sennheiser e906 (which has a switchable LPF). These mics have better low end response and aren't quite as brash on transients. But they have their own quirks which you have to be mindful of so there's no free lunch.
If you can't buy more mics, try adding a second SM57 to cabinet. If you ran both mics at equal volume (same gain and eq settings) you'll get a 3dB boost due to the summing. But if you take one side and apply more subtractive EQ, you'll get a 3dB boost of everything that's left, namely the stuff you want. Or process the second channel separately by adding EQ and compression and then blending with the "dry" track. There are many ways to skin this proverbial cat (no animals were harmed in the writing of this post).
One last important factor: SPL. Dynamic mics are less sensitive and will inherently sound thinner at low levels. On club stages this is never a problem but in today's houses of worship SPL is the new 666. For them I recommend a small, inexpensive iso-cab like the AxeTrak (
http://www.jlhproducts.com/axetrak/).
As always, YMMV, IMO, AARP, KISS
And if you made it to the end of this post, you should win a prize or something.
