you could turn off 'focus-follows-mouse'... the delay, which is like a split second, is due to transitioning the focus from one window to the other.