Some of the spells used will auto break follow.
Channeled spells, and any melee auto swings will instantly break follow.

That was something that plagued me until I figured it out.