I have XP64 too and I used to have 4GB of RAM. I disabled the harddisk swap file(because of performance issues) which worked fine with 4 WoW clients(memory-whise). When I upgraded from 4 to 5 clients I got into trouble because all WoW apps would frequently exit because "there wasn't enough virtual memory available".
I upgraded to 6GB and the crashing was resolved, BUT WoW also started using more memory(700MB+ instead of 550MB+). Windows then used about 2GB+ for caching so I ended up with only about 500MB available. For some reason(memory fragmentation?) my performance dropped down when having this kind of heavy memory load.
Then I upgraded to 8GB and all was fine.

You can run 5 clients with 4GB memory too, but then you have to make sure that you have the swap file enabled (and fixed size and at least 4GB big). The biggest downside of the swapfile is performance, because your PC will often write unused memory to harddisk (which is slow).