Your toggle needs to be in its own keymap. Once you toggle off your current map there is nothing left on so no keys are mapped.

edit: Any toggles that you make should be in a persistent keymap that never gets turned off.

edit2: And don't forget to add those new keymaps to your character sets.