I'd say you're reaching the limit of your ankles endurance, and I wouldn't worry so much about HOW you keep time since you are aware of keeping it.
Your internal clock will improve until you don't need to physically do anything (except play) to keep time, and if you are playing with a group you shouldn't have to be tapping your foot to keep time.
If its really becoming an issue, play with a metronome so when you're foot "drops out", you still have a count behind you.