| So, how about the 3rd, and 5th fret of the same string? The 4th fret may be a little low in that position on that string. I would not do any fret work to fix this if it doesn't bother you. That would mean lowering the others, and you might create a whole new can of worms.
If you don't have the stuff to check for level frets, you might take it to a tech, for nothing else but piece of mind as to where the problem is.
It's not too uncommon to have a little fret buzz on the E string, as the tension is lower, and it moves a little more. If it's only when played acoustically, I'd be inclined to let it go. Some minor fret buzz is not heard through the amp.
Have you raised the saddle on that string to see if you can eliminate the problem? Because of the lesser tension on the bigger strings, the action is usually a bit higher. | 
