In that situation I think I would start a new crew to box four that only groups with the person who plays occasionally.

I box five but am setting up a group of three to play with my brother and his friend.