The US boxing servers are...

Alliance - Kil'jaeden (Legion of Boom)
Horde - Blackrock (But Mom I'm Multiboxing)
Horde - Magtheridon (The Zerg)

Chances are, playing with other boxers will outweight having fewer players on your faction.
Being on the side with fewer increases the chances of world pvp against you.
And having several boxers on your side would be fun.

If you don't have a vested interest in any server, go for a server with other boxers.