If you use Jamba there are settings to auto-deny guild invites. That is stopping you.