Psychologically:
5's are like the heavyweight division. The top 100lb and 150lb fighters wouldn't fair well vs. the top 250lb fighters and the same goes for arenas. We all want to have the maximum amount of power possible when we push a key - that's why some even continue to play or plan to play 30+ chars despite the raid cap being 25 on all the new content.

Also when you stomp another team in 5's with 5 characters of the same name they're forced to deal with the reality of what happened. In 2's or 3's they could just assume it's a couple friends who thought it would be funny to have matching names

Technically:
5's also cross into a threshold where you have the potential to gain a coordination advantage over a 5's team with 5 human players when it comes to avoiding overlapping cooldowns due to the amount of vent chatter required for a 5's team to fully coordinate that. In reality we probably never actually get this advantage since the best teams will use every addon possible to help track the cooldowns of both their teammates and enemies to avoid wasted cooldowns or diminishing returns even more than we can, but at least in 5's there's that slim chance. In 2's and 3's it seems the best teams pretty much already know what the exact gameplan is vs. each comp they are likely to face. Then there's the more obvious coordinated burst advantage, the more the better when it comes to that.