Musical references in HPL's stories include "The Music of Erich Zann":
http://www.hplovecraft.com/writings/texts/fiction/mez.aspx
And "The Festival" where he mentions a "feeble flute" being played by a half-seen, amorphous player as the narrator walks with a procession of unwholesome celebrants of the ancient yuletide festival held under decrepit old Kingsport (fictional) on the Massachusetts coast. Read this one to your kids at Christmas. It is vintage Lovecraft at his best:
http://www.hplovecraft.com/writings/texts/fiction/f.aspx
Second only to "Rats in the Walls":
http://www.hplovecraft.com/writings/texts/fiction/rw.aspx
Although I don't recall this story having any references to music.
Both stories hint at mankind's true origins and purpose. That all you see around you in this everyday work-a-day world is just a shell, mud to be scraped off the bottoms of shoes that ill-adorn the feet of that which cannot, must not, be named. That which squirmed through the primal ooze and crawled along the ocean floors many thousand fathoms beneath earth's bright, sunny surface. And which digs and burrows its way through the fetid soil of ancient graveyards and scurries for darkness from under every disturbed rock. That which walks which ought to crawl.
From what I understand, Lovecraft loved classical music--nothing wrong with that--but detested jazz. Some of his more cosmopolitan friends tried to get him to accompany them to jazz nightclubs but he always declined. He saw jazz as the musical expression of an inferior people. Lovecraft, like many whites of his time, was a racist (remember in the mid-20s, the KKK was 4 million strong) and a deep-seated neurosis that was no doubt instilled in him by his smothering, crazy mother (who died insane in a hospital just as his father had some years before except in the case of the latter it was due to advanced syphilis). But I have always suspected that HPL himself suspected that his racism was a mere narrative setup in his mind to mask his true misanthropic misgivings which his stories persistently gave voice to--the loathsome things we are akin to through common (and ancient) origin. The findings of the mapping of the genome might have both disgusted and yet not surprised him, terrified him but confirmed what he already knew (or, more properly,
because of what he already knew).
And, yes, metal seems to be a natural musical expression of all things Lovecraftian (which, in this day and age, has replaced the term "Kafkaesque"). I wanted to set Lovecraftian themes to music in the 60s and I envisioned distorted guitars, pounding bass, booming drums, a huge pipe organ, a choir and screeching evil vocals. At that time, I was influenced by the Electric Prunes Mass in F Minor and thought the music should be constructed along those lines except the voices should be deep, bass male voices that bordered on the inhuman. After all, what other kind of music is fitting for the King of Gothic Horror:
It's kind of funny that metal has sort of caught up to that musical vision. Lovecraft would have loathed it and disavowed any connection to it but then nothing else would quite be him, would it?