Do you enjoy PvP Arena? You can do that with 3. I enjoy PvE dungeons, for that I need 5. If you enjoy questing, 2 is usually good.

I think you've been bitten by the 5-box bug and you should just go for it. Just wait until the 10-box bug bites you.