A really good question - I'll take an amateur stab at this

Take street fighter 2, where the basic skills for each character were the same, but they then had some special skills to differentiate, whether it's speed, power, range etc...

Then add in the possibility to button mash, some kind of learning curve with complexity, thin women with breasts and metal armor covering useless parts of the body (for fighting, that is), a fat guy, a cool guy, a guy with spiky blond hair, plenty of 19th century morals in relation to gender equality, and maybe someone wearing a robe, eyeliner and a glowing wooden stick to boot.

I'm getting carried away here...

The point about class balance doesn't extend just to MMO's; but it's certainly the most complex as you have more abilities and an open world with some kind of physics which means the combinations are virtually endless. This is why trying to balance PvP is impossible and WoW is failing at it by providing the same talents to every class, thus removing some of the differentiation of why you would choose lock over druid, for example.

To prove this point, you wouldn't choose yoshi over bowser for the ghost circuit 1 in super mario kart on the snes, would you? Horses for courses.

For PvE I would imagine the companies would use a combination of total dps with perfect rotation to start ranking the classes, then provide additional talents for the ones with less DPS, such as utility. Then run a simulation of different classes using optimal rotations to get to a level playing field, then build raid bosses which require half of theses rotations/dps.

For PvP I would run a 1V1 comparison for each class combination taking into account all abilities (including sentry totem) and make sure they stack up. All while ensuring the resources get depleted evenly for each character, whether it's mana, health or bedtime

Essentially it's just an incredibly complex resource planning exercise.