I've never had that exact problem. Sometimes all followers will just stop working completely and I have to switch to each window just to give it focus, then its working fine again.

For me it feels like a button spamming issue but I cant reproduce it on purpose.