In that situation I think I would start a new crew to box four that only groups with the person who plays occasionally.
Good advice.

Well, it must be good because that's what I do

I have a few teams on the go (mainly because it's absolutely soul destroying to play without rested bonus - roll on the faster levelling Blizz have talked about!!!), and while most of them are "just for me" I do have two team specifically for playing with other people.

One of three, which I regularly play with my wife and a RL friend (I just provide the DPS and crowd control - relaxing, if mind-numbing, bliss!!!) and one of four which I only play with 'er indoors.

Works out really well, although I must say that it's really not that difficult to swap characters out when you want to. I often play only two characters for gathering / farming for example.