Pitch detection requires, at minimum, reading half the wavelength of a note. So for B0 at ~33ms wavelength you're going to be more than 15ms to detect the note and then shift the pitch to the new note. Add that to the baseline latency of the DSP system and you get a little squirrely on the low...