There is some trick but I forgot what it exactly was. Check out Totalbiscuit/Totalhalibuts video on Youtube.

Pretty positive you have to make sure you interrupt his spells to be a success.