My best guess is the same reason I lose /follow on some of my chars at random times as well.

You might be losing data packets being sent from your connection. Its not alot but even missing just one will cause the lag you speak of.

I still have times where one of my boxes ( I run 5 from one comp) will be Red Lat and the rest are Green and are <40ms.