Did some profiling and found out that the bottleneck was definitely the GPU. Upgraded to a refurbished 970 and it's like night and day. No issues whatsoever now. Definitely worth a few bucks. FYI I run my slaves at 10 fps, but setting them to 30 makes no noticeable difference as far as I can tell.

So I guess I can put off the system rebuild until next xpac.