So basically, no target can take more then the maximum damage for the AoE spell.
And up to 10 targets will take the full damage.
But pulls in excess of 10 targets will result in less then full damage per cast.

Still, with 4x or 5x toons doing AoE, for example 4x Magma + 1x Consecrate, that's an awful lot of AoE in one area.
Not sure how many mobs at once we get.. with say all green slimes for the Neutralize the Plague quest in Icecrown.
Aside from boosting, how often do we have more then 10 targets in one AoE?

As far as our toons go, most of us will still take full damage from any type of AoE.
But, when we're bunched up with several other players, each AoE will hurt a lot less.