I had a rogue team (4 rogues + druid healer for pvp - prot paladin, priest + 3 rogues for pve). My experiences:

PvP: They really suffer from kiting. 1 well geared mage can challenge you, 2 can annihilate you if they know their stuff. Ganking is hilarious with them though.

PvE: Works pretty well, although melee teams are always more of a challenge than ranged teams due to AE and positioning. Don't see any 5-man content that isn't doable with 3 rogues as dps, but you might have some challenging bosses along the way :-)