I was taught to read music in 3rd grade on the flute, and in 5th grade my teacher tried to teach me to sight-sing, but she was not really able to.
My point is not that children can't learn to read music at an early age -- some can. Rather, it's clear that the process of teaching kids to read music emphasizes a skill that is highly specialized, and which many kids are not prepared to learn in the early grades.
In teaching music, what's the dropout rate? How many kids actually continue with using the skill? I'm all for teaching kids to read music, but it's NOT THE FIRST THING to learn about music. We all learn to speak and listen to language before we learn to read and write it, no?
Well sure, a lot of kids that start playing in school end up dropping out of band or stop taking piano lessons, etc. Lots of reasons for this - music just isn't their thing after all, not willing to put time into practice, don't like the music they're playing, or just a short attention span. I seriously doubt many quit because they can't learn to read simple sheet music played in school bands and orchestras.