I might try using a hardware solution then to send my Keyboard/Mouse control to the 2nd PC on a per need basis. This way if I want to manually control the 2nd PC, I shouldn't have a problem.

Hrm, are there USB based KVM switches these days? Because that's what I got. USB Keyboard and mouse. Hrm.