1 ms = speed of sound for 1 ft
So 55 ms would be like standing 55 ft from the speaker.
If you're doing VST effects then this may be some trouble.
If you're just recording and not listening to the processing then it can be a bother.
I agree, that seems high, and 10 to 20ms would be more comfortable (same as 10 to 20 feet from the speaker)
As get mentioned many times, Start with CD quality to record. 44.1khz 16bit
And your computer works less. Set the buffer to a smaller size and see what this gets you. Play with these settings.
Read the dpc latency checker page top to bottom, it could be something else in your computer using up the CPU.
http://www.thesycon.de/deu/latency_check.shtml