I kill the end boss without clearing pack's. Jump down from the top and make sure you fight him behind the pillar where the imps come out of the cave,
make sure your minions are right in front of the cave to stop the imps from running out ( this is what agro's the packs of flowers)
easy as