Of course, if you are able to thoroughly clean the surface with an appropriate solvent that actually binds with the silicone rather than just smears it around, adhesion will be ok, but solvent options are more limited in CA where CARB has placed limitations on what solvents can be used without controls and/or permits. Xylene is one of those CA restricted solvents because it's benzene/toluene based and derived. Doesn't mean that it's not available, but that it's difficult to source or may be formulated differently for CA. The same thing is true of Lacquer thinners, I can no longer get anything but CA formulations which are distinctly different.
On very convoluted (textured) surfaces, it's also difficult to remove adequately because residue will remain deep within the texture.
Where the adhesion issues will show up is under abrasion tests where both downward and sideways force are applied simultaneously. These are the types of tests I apply to all coatings to minimize the possibility of coating failures in the field.
The thought of using spackle or drywall mud, or anything like that is frankly absurd. the strength of any coating is governed by the toughness of the base material, base material uniformity, base having adequate porosity and mil thickness of the coating. The difference in force toughness between spackle and either the underlying wood (birch) or bondo is like 100x weaker. This is rule number one when it comes to finishes is a properly suitable base. (I spent a lot of time in the metal and wood finishing industries years ago)
IME, the difference between a true catalyzed polymer and Duratex is pretty huge. Unfortunately, so is the cost and difficulty in application (to do it right).