Since it only happens randomly I think the problem could be lag. Very often I will hit assist and the followers still think I have my previous target but hitting again a second later works.