You mention "an Action Target Group" but since ISBoxer 38 you can type ad hoc groupings (such as multiple ATGs) into Target boxes as well. http://isboxer.com/wiki/Target

The order when you use the Round-Robin check box depends on the order of the characters within whatever "set" of Targets is defined. For any given ATG, that order will be related to the order they joined the ATG.

If you need to guarantee that it happens in a specific order, then you will need to be more explicit with the Target -- for example by specifying each Target in the order you want. Say you have an ATG that usually has the first 3 slots in it (is1 is2 is3), these characters are joining or leaving this ATG dynamically, and you always want something to happen in the order is1 is2 is3 for some purpose. You can put in the Target box "is1,is2,is3" (no quotes) and then the Target is going to be the set of is1,is2,is3 (consider it like a temporary ATG) and in that order for Round-Robin. You just lose the dynamic nature of an ATG. The Target can also include or operate on multiple ATGs, e.g. "is1,is2,(paladins&dps)" would go is1, is2, then any Characters in both paladins and dps ATGs, noting that duplicate characters are filtered out for round-robin so it's okay for is1 and is2 to be included in paladins&dps and they will always go first in the rotation.

If you don't want any filtering of duplicates, or have another reason not to use the check box, you can also manually implement round robin with multiple Steps in your Mapped Key instead. Each Step can have its own round-robin as well.

One last note for the round-robin checkbox, if you have multiple Actions in the same Step they are going to use the same counter for round-robin, applied to their own Target(s). So if you have 3 actions that all use the Round-robin check box, and they have the same Target to round-robin through, they will apply to the same window.

This is probably way more information than you hoped for, but hopefully that gives you the information you need to decide whether to use this box or another method