Check the actual connection latency on that green bar by your backpack. I've noticed that < 150 everything is fine with follow, but once it goes up higher than that it can get sketchy, and >200 means I'm going to lose follow on my follower if we're mounted and I don't slow down occasionally.