If you use ElvUI, there is an option in the General tab to only accept invites from Friends/Guildies I believe. If you don't, not sure.