My follow doesn't seem to work unless the toons are grouped as well.
It might be an IS Boxer thing.

Same battlegroup or raid works fine; I've not been in non-BG/Raid situations where my "set" is not in my group.