Its not entirely clear who IWT is being spammed on. I read this as you having all 5 characters doing hand to hand and then you want one of them to cast a spell. When you do this there is a delay. My guess is that your queuing up these interact with target keys that are being spammed and it takes a while for them to bleed off.

This seems to be why characters run off in crazy directions as well, that during this queue they are apt to select a target if one is not available or similar. That there is no way to bleed off the IWT.

I believe this is more a function of the mouse click to move pathing than actually targeting.