Honest I can't tell if you are saying my thoughts are worthy or unworthy?
Here's a page on color theory
Primary Colors of Light and Pigment | learn.
Regarding the premise of associating sound with pitch. I think the best thing you can do is make your own system, or perhaps you could devise an experiment and use statistical modeling to determine the general populations central tendency on color pitch association. Unfortunately I am almost certain the results will not show statistical significance, especially since their are so many pitches and so many shades of color.
Maybe a simplified experiment would be more worthwhile. Color and pitch are both frequency dependent. Red is at the low end of the visual light spectrum and violet is at the high end. So you could run an experiment to see if even the frequency relationships hold true. For example, do people associate red with low frequency sounds and violet with high frequency sounds? If this basic relationship does not hold true, then there is no scientific basis...but there can still be an artistic basis...obviously since it's been done before.