In my experience, load time is heavily influenced by the number of addons being loaded.

I've rationalised my UI down to a standard set of addons across all five of my toons and now the load times are much closer together.