Some blocks of USB ports are nothing more than a USB hub. Thus one device on the hub can steal bandwidth from other devices on the same hub.
Follow the USB connector from the connector on the front or back and make sure it connects to the Motherboard - 1 to 1. One PC I have, all four front ports are merely a hub off one MotherBoard connector. The back ports are separate to the motherboard.
For example if you have your webcam plugged into the front USB ports, then plug the M-Audio into the back port and try it.
Try the DPC latency checker
DPC Latency Checker
Find the forum sites where they discuss the results of running this to ways to better your PC performance.