Before you decide on anything, make sure the video card is compatible with OSX, that there's drivers, etc.

I don't know how much support is out there for high end video cards on OSX. I'm a Mac novice, but I'd think you'd have the most success with a card that's been used in Apple manufactured products, such as the Nvidia 8800 or 9600 series or the ATI 4850. Apple isn't exactly famous for using bleeding edge video cards in their systems, so I don't know what your luck would be with something that isn't actually used by Apple in their product line.