If you want to use your 4GB go for Vista 64 imo. XP 64 is nearing it's EOL anyway. Plus driver-support for XP64 has always been meh. As for performance I doubt the average user will notice a huge difference in 32 vs 64.

You might have the odd app that doesn't work on Vista 64 but that's probably more due to Vista then the 64-bit part.