If you have mouse broadcasting enabled, it will always break follow after a few steps. Make sure that is off, and set up a "hotkey" for when you need it.