A person with Absolute Pitch (the Psychoacoustic term for perfect pitch) lives in a technicolor world that those without it cannot imagine. Sorry to be so blunt, but if you don't have it, you have to work a lot harder to do many of the same things, and there are useful things you just can't do that those with AP can.
People with AP and a thorough training in music always have more information immediately available to them as they listen, perform, or compose. It makes a difference. Yes, there are drawbacks, but they are minor compared to the advantages. The impact of these drawbacks is usually felt more by those with AP in professional performance situations where tunings are modified to fit ideas about early music, though that is not the only area. AP is a serious handicap for an ear training teacher who wants to actually help students without AP.
AP is genetic, according to most researchers. Yet, there is no doubt those without the particular differences in brain structure that enables AP can acquire a degree of AP. I have personally experienced it. During the two and a half years I was taking tons of Ear Training at a New York conservatory while also teaching even more ear training at another school--I was also practicing piano 2 hours a day, composing each day, AND teaching other college classes in music--I noticed something surprising happening to me.
I started becoming very sensitive to pitch and key. The musical form of classical pieces I taught and studied started to unfold in new more dramatic ways. Key areas were noticeably different sounding, and this was not just based on obvious modulations or intervals between key centers. D minor sounded like D minor, Ab major just sounded like Ab major. No colors, no smells, not weirdness, just knowing.
I usually knew what key something was in instantly when I heard it, and nearly ALWAYS could identify a pitch played on the piano. The funny thing is the same thing was happening with the metronome. If you had asked me to produce common classical tempi like quarter = 72 or quarter = 96, I could do it dead on. After a couple years of this kind of intense teaching and study, I finished my Doctorate and was also done (for a while) with all the ear training study/teaching I had been doing.
After a few months of less practice and teaching, the AP effect faded. I have had a number of conversations with others about this, and it is not unusual. However, mine was not like the AP that others I have known with "real" AP possess. Later on, when I again started teaching Ear Training, playing a lot of piano, and composing, the temporary AP effect came back. It then mostly faded when I again moved on to other, less musical work.
Much of the research I have read (citations needed here) leaves one still wondering. The multitude of (physical) neurological studies I've seen support the "genetically inherited brain structure" model of AP. You are born with the capacity or you are not. Yet, some psycho-linguistic studies have indicated that Mandrin, Nederlands (Dutch), Cantonese, and other languages encode an amazing amount of AP which native speakers thus ALL posses. This argues for a more environmental/cultural side to AP. The science is just still unclear.
To me, what is clear is that you are born with a brain which is yours to make the most of--or not.
In closing, I'd like to add that I had the great privilege to have been a student and friend of the great American composer, George Perle. George is the finest musician I have ever known. He not only posses AP and a truly virtuosic level of pianistic skill, he is also one of the greatest music theorists AND musicologists of the 20th Century. There is seriously nothing that George does not know in intimate detail about the classical music of the last 300 years.
When George Perle hears a piece of music, I can attest that he instantly understands the deepest relationships that exist between each and every tone in the music he is hearing. This helped him instantly spot all his stunned composition student's errors in tonal and post-tonal composition without fail.
George composes (he has won a MacArthur, a Pulitzer, written a huge amount of orchestral and chamber music) and has written nearly dozen incredible books. Did AP matter for him? Yes, totally.
In fact, a number of years ago, he complained bitterly to me that his AP was "sinking." He was hearing things transposed downward a semi-tone. It was very distressing to him. I had read in several places that as the brain ages, this is common for those with AP. I don't think he was reassured when I told him. It was just too important, and he wanted to find a way to fix it. The fact that in his mid-80s, it still really mattered to the most brilliant musician I have ever known that his AP function correctly tells me something about AP.