This isn't quite the answer you're looking for, but I 5-box on one Mac Pro (8-core 2.8ghz Xeon) and can use the built-in movie capture just fine (I think it's still only built-in on the Mac version). I only capture on the main, not from the clones, and my capture resolution is 960 x 600 at 30fps, works great. Main toon play resolution is most of a 30" monitor (so, something like 2400 x 1400) while the clones are essentially minimized with window sizes set to something like 1024 x 640. I don't know exact window sizes as I set them by hand >.< but those are close.

I did the /console setmaxfps command to limit clones to 20fps, main gets 60 most of the time. Mileage varies greatly while in Shat.

Hope that helps.