I suppose in theory it could be hard drive access times. If you are alt-tabbing between the window you are running in full screen. When an application losses the focus with DirectX it generally has to consider all of the graphics memory to be 'Gone' so it would have to pretty much dump everything and reload it when the window was active again... This would presume to mean a bit of Hard Drive time depending on the situations.

Another thing to consider are the mods for wow. I too had around 30-60 second load times for WoW just for hopping into Stocks or the like. This was due to me running alot of database/collecting mods that simply are not needed on all of the accounts in a multibox setup. This is things like Gatherer, Auctioneer, Enchantrix, LootLink, even Quest Helper (But you want this one no matter what, just set the slaves Performance (Perf) levels to 25% or lower on the alt account(s)). A quick way to see if its the addons is to simply unload them all and log into the game, see if your load speeds and alt-tab speeds are affected