I don't feel like scaling back helps anything. I always still have the 5+ characters to play, so even if I leave them out I still have them sitting out missing the quest and slowing my DPS. But I don't ever find having 5 to be an annoyance at all, either