I have no issues with a pet class targeting a totem and destroying it.
If the warlock/hunter uses their own toon to destroy it, via a macro that's fine too.
A GCD for a GCD, so to speak.

If a shaman can drop four at once, on a single GCD, then maybe AoE can destroy 4 at once, provided the totems are the target in which case only the totems are hit.
That way its still a GCD for a GCD, and the class has to manually target them, not auto destroy them without thinking while concentrating on something else entirely.

I just don't like the pets destroying totems, in priority order (or any order) with zero thought on the hunter/warlock's part.
It would be the equivalent of a shaman being able to.... /cast Lava Blast and having that spell purge 3 buffs which the shaman could choose before hand in a priority sequence.
Well.. maybe not three at once, but one spell purged per keystroke of the dps spells... LB, FS, LvB etc, in the priority sequence.