In short.
No simple rules.
There are rules that are actually quite simple:
- in the context of a diatonic heptatonic scale, diatonic notes are spelled such that each of the 7 notes of the scale get a different letter name.
Example: if the scale of the harmony of the moment is Ab melodic minor (to be played on an Abm6 tonic harmony), the notes would be: Ab, Bb, Cb (not B), Db (not C#), Eb, F, G.
- There is an exception to the above rule (in my view): when using the altered scale on altered dominant chord. For example for a G7alt (resolving to Cm), one may use the G altered scale, which consists of the notes of Ab melodic minor. In this case, however, the B (i.e. the major 3rd of G7) should really be spelled B and not Cb, so that we have both a Bb and a B in the scale (and no letter name "C"). The B is the 3rd of the chord, where the Bb is the b10 extension (almost universally, and improperly, called #9).
- Similarly, when using a tritone substitution, as such device relies upon enharmonic ambiguity, it is acceptable to use some enharmonic equivalent spelling. Example: if one uses Db7 as a tritone substitution of G7, it may acceptable (or even advisable) to spelled the Db7 chord as Db F Ab B instead of Db F Ab Cb, to emphasize that it will resolve to C or Cm.
- chromatic passing or appoach notes are spelled such that the chromatic note has the name of an altered note neighboring the diatonic note it resolves to.
Example: in Ab melodic minor, moving chromatically from Db to Eb: Db D Eb; from Eb to Db: Eb, Ebb, Db.
- Many people prefer to avoid double flats or sharps in spelling passing or approach notes, and use enharmonic equivalent spelling instead, and this is generally considered acceptable; doing so for diatonic notes, however, should really be avoided in my view, as it completely obscures the harmony.
- then there is the context in which the scale of the moment is simply not a diatonic heptatonic scale, so that one cannot ascribe a letter to each note of the scale (atonal music, octotonic scale, ...). In such case, there is no other rule than trying to be as readable as possible...