Well, I *think* I have sorted the problem, and will post it here to let anyone else having this trouble know:

On all of my character sets that I have the background windows throttled down to 10fps, when I switch screens I get that lag after 20-30 minutes of play. On the character sets that I have the background windows throttled down to 30fps, I have yet to get any lag when switching, even after a few hours.

Completely counter-intuitive, but that's the only difference I can think of between the two set-ups. Guess I'll boost all the other sets to 30fps when in background and see if that really fixes it. Here I thought that setting the background FPS to 10 would actually make things more responsive.