Try lowering your background and foreground fps limits to 30. You are using the same CPU strategy that I am. I am running a far older i7-920 (one of the first gen ones) and a GTX 680. So my GPU is probably slightly better than yours but my processor is far worse. I run 24GB of memory though that shouldn't be the issue (I only have that much because of trying to run 5-6 copies of other games would fill up my previous 12gb of ram and I bought my ram in kits so I just picked up another 12gb kit to pair them).

Some other things you can do to increase performance is to lower your D3 settings. Turning sound channels down, settings the Low FX option on and whatnot. I don't care how super pretty D3 is because even on the best settings it is still not a realistic looking game, so turning down everything to minimums is fine for me. Your preferences may dictate otherwise.