I was mistakenly thinking the force would increase as you go towards the centre of the Earth as it's inversely proportional to r squared. But your point about the mass is a valid one. The precise details of how this would work inside the mass of the Earth is trickier to calculate - details are here.
http://hyperphysics.phy-astr.gsu.edu/hbase/mechanics/earthole.html
http://hyperphysics.phy-astr.gsu.edu/hbase/mechanics/earthole.html
