Could you post the macro you have tried? Something like this maybe.

Code:
#showtooltip
/cast [mod:shift,target=focustarget,help] Heal; [mod:ctrl,target=focustarget,help] Other Heal; [target=focus] Nuke
A few possible problems:
1. Spelling errors.
2. Make sure the spells with modifiers comes first in the macro.
3. You could have shift+5 bound to something else already.
4. Sometimes there are problems when cutting and pasting. Maybe some tabs or spaces at the end of a line that you can't see.