You need to have your minions on /follow

If you are using directional keys to try to control all of them, it is pretty much impossible.

I have my minions on /follow and use my mouse for movement.

I have the directional, if I need to move them all forward/backward/strafe left/strafe right which is situational.