It's latency.

If you look at your alt screens, they'll appear to be closer to your main than they do on your main's screen.

Whenever you move, that gets sent to the server. It then gets sent to everyone else's clients (including your alts) so they can update their displays.

If they're on follow, they then move and tell the server. Which then tells everyone else (including your main).

What you're seeing is the time it takes to send all those messages and process them.

So you're stuck with it.