The guards on the isle are broken. there are a ton of places you can stand and attack without the guards going after you. Its a horribly broken area. :-( Nothing really you can do except not to retaliate and to move away from the guards first with hope that the opposing faction will follow.