Enabling v-sync on all my toons solved the problem, both the followers and the followed. Also, get Fubar and PerformanceFu. You can force a garbage collection when your memory increase rate gets too high, as is the case with this follow bug.

It worked for me so far.