I do think I've heard that some magnets, Alnico and whatever keeps their magnetism well into heat, but dimishises when it's really cold. But if you have had it in cold it doesn't lose magnetism for good. Letting it slowly creep up to normal room levels restore their magnetism without problems.
Ever heard of cryogenic treatings? They do that with magnets of pickups, steel bridges, strings and so on. It's pretty down in temperature that you can't get in nature at least, not on earth. And they bring it SLOWLY back up again, and the structure of the metallic changes a bit. For a more durable reason moreso than anything else. But there's other things to speaker cones that just the middle coil. The paper and everything else can shrink or dry out, and you can sure make a crack easier in a paper cone.
But as people has said. Car radios speakers have been in the most extreme settings, both at sauna levels, and well below fridge or freezer levels but still works. I think the speed of getting cold or hot is the most important thing, and even more so humidity. Even english manufacturers Tannoy does outdoor speakers with coaxial structure and the last until hell freezes over.