/grumble.
It is def the keymaps that control High and Low settings. I just built a brand new team through the wizard, isolated all the keymaps, and associated steps and actions.
I can have keymaps enabled (minus High and Low keymaps), it's perfect. The moment I bring the High and Low online, swap lag issue.

Just to confirm, everyone that does High and Low keymaps:
High is set to Window: Current
Low is set to Window: All w/o current

then on your swap keymap, you just enable an action to "do". Both are actions targets are Window: Current.