I am going for druid + priest + mage + warlock + warlock.

I intend to use some hotkeys (probably F1-F3) to change functionality of my hotkeys.

A tank mode. (used for dungeons and maybe some outdoor elites)
It should cause hotkeys to do dungeon functionality, so druid will go bear form and tank and healer will heal and mage/warlocks will have some frostbolt/shadow bolt button and some aoe button.

A healing mode. (likely used midfight in world pvp and swapping between healing and dps mode depending on situation)
It should cause druid and priest to focus on healing, they will likely idle mostly in this mode when they don't heal. Mage/warlocks should be able to dot, frostbolt/shadow bolt, aoe.

A dps mode. (likely used outdoors when killing mobs or when enganging other people in pvp)
It should cause everyone to use dots and aoe. Maybe also everyone able to use cast time or just mage/warlocks.

I think healing and dispelling won't really be any issue.

I am not really sure how I should effectively use 2-3 dots on warlocks, 2 dots on druid, 1-2 dots on priest, cast time spells, instant spells, aoe, cds.
There is so many good abilities and so different in how spells are cast that it will be really hard to do it effectively.

I think I will use a button with corruption/Pain/moonfire/frostbolt and another for curse of agony/hex of weakness/insect swarm/frostbolt.
Probably should add another 2 dot buttons for curses, 1 for curse of tongues and 1 for curse of weakness.
Probably need 4 buttons to handle dots.

I will probably have to use Psychic Scream/Howl of Terror on round robin and reset to Psychic Scream every time it should be ready, so I generally don't have to use howl of terror but can if needed.

I might do the same kind of round robin with War Stomp and Infero (with @player on inferno).

I could make it so druid uses Hurricane with aoe button in dps mode and Tranquility in healing mode and let it heal normally when it is on cd. While the other characters uses arcane explosion (maybe blizzard in dungeons), holy nova and hellfire.

Going to need a lot of buttons for offensive dispel, silence, fear/polymorph, cold snap, frost nova, death coil, blink, life tap, shadow burn/fire blast, maybe bear form/bash/feral charge, innervate, WotF and a bunch of other abilities.
Probably need some buttons for pet management also.
Also going to want buttons for goblin sapper charge and some useful trinkets.

Most complex team I played before was dk + 5 pala + 3 druid + 1 priest on wotlk PS. I feel like this is going to be 10x harder.