I ran in to this plenty as well. Even had it happen when there really seems to be no one from the opposing faction around. E.g. I've gotten flagged for PVP combat just walking around in the horde capital. No idea what causes it, but as JG suggested, I also found that pet classes have it happen more often.