I believe the casting debuff is worse the closer you are to him. If you stand at max range it only increases cast speed by 50% I believe.