Does the delay clear up when you disable all your addons? Addons usually write new settings when you log out, but only when you do so properly, as you see. Addons with huge databases that they have to write can further extend your logout times while it does its thing.

As per Blizzard's standard troubleshooting :

delete the Cache folders located in World of Warcraft and World of Warcraft\Data and then move the WTF and Interface folders to your Desktop


http://us.battle.net/wow/en/forum/topic/3229085244

I also stumbled across this, but I am not the one to talk to about such issues.

http://code.google.com/p/arkinventor.../detail?id=452