I'm guessing that you have focus-follows-mouse turned on and that you use the mouse to steer.

If that's the case, as you move the mouse around, it drifts into the other windows (even though the cursor is invisible while you have the right mouse button held down) and so the movement keys get sent to the alts.