I have bound bottomfightactionbar 9 to F! just as a test and it would cast the first step (nomodifier) totem but alt+f1 and shift+f1 do nothing
have you tried to use other keys as F1 ?
Cause for the function-keys (F1,F2,...,F12) not all modifiers are available.
See: http://www.dual-boxing.com/wiki/inde...ld_of_Warcraft

In this case 'alt' as modifier would not be available.
Onother thing you could try is to get a all-in-one castline like this:

/cast [nomod]Fire Nova Totem;[mod:shift]Searing Totem;[mod:alt]Magma Totem

(but i'm not exactly sure if this helps)