I'm using Grid + Grid Custom Layouts.

/Grid Config
Layouts > Custom Layouts > New

Use this config:
ToonA,ToonB,ToonC,ToonD;NAME
1,2,3,4,5;5/5;NAME;NR

Line 1 corresponds to the leftmost column.
Enter every toon you have, in the places of ToonA through ToonD.
The format is Toon -comma- Toon -comma- Toon -semi colon- NAME
There are no spaces.
I have 41 toons listed on this line, every toon from every team.

The second line specifies, only characters in the first five groups are to be displayed (1,2,3,4,5)
They are to be displayed in a five column by five row Grid.
The sorting is alphabetical, the same as for column one.
And NR = No Repeating, which means if a name appears anywhere already (such as your toon name in column 1), it will not appear in another column elsewhere.

End result.
We get our five (or ten) man group in Column A.
And the rest of the raid/battleground in Columns B-E.

You could tweak this to:
TankA,TankB;NAME
DPSC,DPSD,DPSE,DPSF,DPSG;NAME
HealH,HealI,HealJ;NAME
1,2,3,4,5,6,7,8;8/4;NAME;NR

Which would display your two Tanking toons (use their names, not Tank) in column 1.
Your five DPS in column 2.
Your three healers in column 3.
Everyone else from groups 1-8, appearing in a Grid up to Eight Columns wide, with no more then Four Toons per Column... in Columns 4-8.