I have almost the same hardware as you but i have 8GB ram and windows 7 RC... My load times for 5 accounts with IS is less than 30 seconds to be logged in.

I would guess you got something else wrong besides IS or a setting in IS really messed up. If you make a new team do you get the same performance issue?