I ran in to this plenty as well. Even had it happen when there really seems to be no one from the opposing faction around. E.g. I've gotten flagged for PVP combat just walking around in the horde...