Happens to me too.

All I do then is switch to one of them, hit my follow button then switch back to my main and carry on like normal