I am dumb. Enabled V-Sync, I did, but not via my graphics card control panel! Duh, of course it would not work!

Over 1000 B/s outgoing bandwidth and no elevator in Azeroth or Outlands will disconnect my follower now. Time to play

(edit: Still disconnects when around 1200+ B/s outgoing, so garbage cleanup has to be done from time to time. Let's see if it can be macroed as well!)