Running vista I am always sitting at 3.2gb of ram used (out of 3.5gb available to the OS) on my Quad Core 4GB system.

3.0gb ram will 4 box nicely, but adding a 5th will start paging to disc, and you will suffer.

I'd recommend a 512mb video card as well, 3 clients will fit nicely under the 256mb cap but over 3 and you'll start using system ram for your video graphics textures and see a performance hit.