That first batch of Garrison quests is a real pain in the ass. Once you complete the Garrison invasion quest, you can ignore the other garrison quests (those that you obtain from NPCs inside your garrison) IMO. I do occasionally pop in and send guys on missions but that's about it. I've been working on world quests since then and once I got out of that first zone, things sped up pretty nicely.