Yeah - ATI and NVidia don't mix well. The problem isn't so much the physical cards, but the drivers. Neither wants to have the other driver installed.

Also for 3 monitors on the 5870 be sure that one of them is display port. You can't do HDMI and 2x DVI. Apparently the HDMI shares the internal connection with one of the DVIs.