After a lot of playing around with the settings, it seemed to have something to do with CPU setting in Isoboxer.

I had set main window to 70 fps and for some reason it was dropping my fps like crazy. When I reset it to the default of 60 I could play again no problem.