I don't know if you need this explanation or not but the thing about counting is to anticipate where the beat occurs and play simultaneously (or at least in time) with the next beat. We don't count one, two, three, four, one then play; we play on the one, getting the feel for where it (one) will fall by counting.
Lots of things will throw off accurate counting of a beat or tempo but they can be worked through and eventually overcome. Using a metronome to practice and play with is one way, learning to identify the time cycle of a phrase in a song is another.
If getting by most of the time is enough for you, great! If you want to be a better musician, and feel working on time will help, fantastic. If I have gone up the wrong tree, I apologize.
Check out Sister Moon on Herbie Hancock's Possibilities or Filthy Habits from Frank Zappa's Sleep Dirt and try counting and feeling the beat.