This very occasionally happens to me as well.

I don't pass ANY movement keys to my followers at all, so I'm not sure how they're doing it, and it's so occasional I haven't been able to find a pattern to it yet either.

Basically what will happen is a single follower will break follow and go running off in the direction they're facing.

Why it happens is a total mystery to me (and it's so infrequent that it's not a real problem - once it wiped me, but ah well).