The level difference makes a big difference at some points.
As for the shouts, try moving behind him when he casts it, i think that should soften the blow on some of them.