Knuckle_head's gauge recommendations are excellent.
The highest string will be a plain steel, some guitar plains are just long enough (Rotosound, D'Aquisto) to reach the tuner post if you have a 2+2 headstock, just thread an old bass ball-end onto the string.
There is a
very rough gauge formula for roundwounds, derived through physics:
2 strings tuned a fifth apart on the same scale will have very roughly equal tension when the gauges are in the ratio 3:2.
Low to high: multiply gauge by 2/3. High to low: multiply gauge by 3/2.
When going from roundwound to plain or vice-versa the rule doesn't work since plains are a solid bar of metal and therefore denser. The gauge of the plain steel string should be lower than the rule suggests.
You can also use a tension chart such as the
CircleKtensionChart pdf to create a set with equal tension on each string.
Don't worry about the nut slots, the downforce at the nut should centre the strings in the curve of the slot floor.