Big subject for a game of a few thousandths that make a huge difference in playability.
Clearance: Fret the string at the third fret. There should be roughly .003" clearance between the string and the first fret. If you do not have feeler gauges a sheet of notebook paper will do in a pinch.
The quickest way to get to depth is to stack up feeler gauges next to the nut. How much? Add up the height of the fret from the fingerboard to the top of the fret plus .010". Stack the feeler gauges next to the nut. Cut with the file until they hit the gauges. Now you're near final depth. From there it's all about feel. A few strokes and test at pitch. Be careful. It is easy to go to far.
Yes there are dedicated nut files. Some are
single edge toothed files. Others are
double edged.
Caveat: This is not a job for the casual repairman or want to be. As always, if you must hunt through the kitchen drawers to find your tools, still haven't memorized that "lefty loosey" is counter clockwise, or think that changing a tire is working on your car, please take your guitar to a professional.