Dump 'taskmgr' and use perfmon or at least resource monitor so you can break out the processes band understand what their actual usage is (real/swap, where the growth is, etc)

This 'after a while' stuff is where i would concentrate my time, if just sitting in an instance grinding for hours by yourself and you run out of memory, or sitting in dal. Look at any of the number of good ace addons that track addon performance, my guess is that something is in a race condition and isn't garbage collecting (x5). Simply showing the top memory growth addons should expose this.