IWT seems very weird to me. When I'm close to a target but not in melee range, using IWT does nothing. Is this normal?
If Click to Move is not enabled, IWT will not move your characters. If you're playing a wand class and are using a wand, then it will also not move your characters.

You didn't specifically mention if you started using ISBoxer or not, but I'm going to assume you are because you mentioned "mapped key" in your post. The IWT Mapped Key turns on CTM for you. If you added IWT directly to your combat buttons (e.g. the 1 button) by using a keystroke to just send the IWT key binding, then CTM is not being automatically turned on by this. Instead, use a Do Mapped Key Action to perform "Interact with Target - Keep CTM" in the Non-combat Key Map. This will enable and leave CTM on for you.

Also, sometimes in combat, my characters start running in random directions. Based on what I've read, using IWT might cause this. Are there any ways to prevent it?
No, it is not preventable unless you don't use IWT. This is by design of IWT+CTM -- your character will run straight at where the target is when you hit IWT. If the target moves and your character is unable to interact because of it, then your character will continue moving in that direction until you stop him. Think of it like autorun turning on and then it will automatically turn off when you IWT -- autorun won't turn you, and it wont turn off if there's no IWT. Discussion, videos, etc at http://isboxer.com/wiki/WoW:Interact_with_Target

How does looting with IWT work? I have auto loot active on all my characters but I still have to manually click to loot the quests items, otherwise only the leader gets loot. Spamming IWT doesn't work either.
Spamming IWT is how I've always looted quest items

I use round robin on some abilities(mass spell reflect, indimidating shout) but it doesn't seem to work very well. The abilities don't seem activate very often, even though I press the mapped key and I have the abilities bound to same keys on all of my characters.
If you used the Mapped Key Wizard to set up new Round-robin Mapped Keys on any of 1 through =, then the key may be double-bound. If you're using the Standard Menu, then you can modify the appropriate button in the Button Set "Standard Assist+Broadcast" by checking the round-robin box on the Keystroke Action (e.g. 1 -> all, as opposed to Assist Me -> self), and drop your round-robin Mapped Key. (Select Standard under Menus in the top left pane, then the button set in the bottom left pane in order to configure the button behavior)

Is there a way to activate mouse broadcast for certain abilities? This would be really good for abilities that require clicking on the ground(heroic leap, black ox statue, mass dispel etc). Using heroic leap is really awkward when you have to use the ability, activate mouse broadcast and then click on the ground.
Using the Mapped Key Wizard, select the "Straight Key Broadcast Plus Click" type of Mapped Key. This will send a key and also temporarily enable broadcasting until you click on the ground. Use the Hotkey you activate the spell with, then go into the Key Map you put it in and adjust the new Mapped Key to use a different Hotkey (e.g. include a modifier like Alt or Shift or Ctrl). You don't HAVE to change the Hotkey, but as with above if you are using the Standard Menu and your AOE spell is on one of 1through =, then this will end up double-binding that Hotkey.