Some decades ago a company marketed elliptical guitar strings. I don't recall the name. Supposed to be easier on the fingers. Nice experiment. Gone in a flash, probably for the reasons stated above: you can't get a string to wind and lay flat, since it will want to twist. And let's just let the nut complications go. There is a company called Apex which has what they call elliptical guitar strings, but it is the windings that are elliptical and wound edgewise around a conventional core. Of course, that retains the circular overall cross section of the string.