what are you showing as latency on both connections?
because normally i have between 20-40ms on each connection (Virgin 20mbit Cable) but if I'm downloading something (as well as playing 5 characters) that can raise to 200-400 on each toon and i get the symptoms that you describe and sometimes one of my followers will randomly drop from /follow