Here's my current setup for my 5-man.

3 toons on my main computer (no problems here)

2 toons on my secondary (the one with the problem)

Hardware:
Dell Optiplex GX270
Pentium 4 3.2Ghz
1 Gig of Ram (should upgrade to 2 gigs)
NVidia GeForce 6600 GT - 128 MB

Windows XP

WoW Settings:
The 2 toons are running at 640x1024
Lowest possible settings for both
MaxFPS 25

The problem is that for the first hour or so everything is fine. (25-30 fps, ~100ms)
After that hour or so, it lags pretty badly and the buildings disappear. (2-4 fps, ~100ms)

I tried logging out and logging back in, but the problem still shows up immediately.
I also quit WoW completely, logged back in after 10 minutes and the problem still shows up immediately.
Rebooting the computer and I'm able to play again.

I'm getting an upgrade for the memory to 2gb. Hoping that this would be the fix. And hoping that I don't have to buy another video card.

Any thoughts?