So, I had ISBoxer assign the clients to a seperate core and now everything is as it should be.

I also capped the main window to 25FPS and the background ones to 10FPS and it's a lot smoother now.