After a little messing around. I unchecked everything under the invitations section. That worked for me. So no accepting friends, guildies or members. I can still send the invites now. I just have to manually accept them all. Which is ok since I still have my big Accept button for quests and stuff.