Do you know what kind of paint it was on there? A wood grain filler probably isn't going to bond very well to that paint down in the grain. I wouldn't risk it. And you'll have a lot of trouble getting that primer out of the grain. You'd have to sand off a lot of wood.
At this point, I recommend that you brush paint the whole body with a dark brown polyurethane, to cover the color of the primer. Let it cure, then gently sand it back, removing all the brown on the surface and hopefully leaving only brown down in the grain. Touch up and sand back spots if needed. You want to end up with only the grain filled dark brown. If you can get there, then you can start wiping on coats of Tru-Oil.
But it may not work. The primer may have filled the grain up to the surface. If so, when you sand back the brown, it will just take all the brown off. The problem is that you want to change the color of the primer in the grain. And that's hard to do. You can't stain it. You have to lay down some brown paint on top of it.
