Ohh I've had this happen also.

My four slave toons will disappear from the main toon's view. But they are still there, just invisible.

Spooky.