a lot of times i find this happens due to lag. i've done this before when i target a mob to check it out and then target back to another mob, cast, and i get my slaves attacking the previous target. so maybe wait like... another second before casting.