Jubb happend to me too.

My solution is putting mithril squirrs on the boosts of the dudes that r following ya, atleast that prevents them from falling behignd