Keep an eye on your followers when rogues are around (moreso than when you don't know they're around). Distract will break follow on one or more of your guys from time to time. Annoying as hell and lethal for the poor guy if you don't see him and ride off. :P