You can check for your combat status, but you can't check for your pets combat status. You can check for your pet's target though.
/petattack [target=player-pet-target, noexists] (this assumes you have a target, so first condition)

The problem with mouseover is it is part of a target, so can't check pet target and attack a different target without getting fancy. So might just be safer to just assume the pet has put you in combat.
/petattack [target=mouseover, exists, nocombat] (in other words, pet won't go if you are in combat, should be 2nd condition)

/petattack (just makes pet attack whatever you have targeted, so can put in empty conditions [ ], should be 3rd, no reason to check for combat, since it is the same as 1st condition)

/petattack [target=mouseover, exists] (should be 4th, again, no reason to check for combat, since it is the same as 2nd)

1st and 3rd also 2nd and 4th are the same (unless I am missing something) If you have a target, make pet attack, sure you can check if on a Tuesday, but the result is the same. If you have a mouseover, make pet attack, again, you can check if the target is a humanoid, but still same result.

Why not just use the keybind for petpassive (CTRL-0)? Just push it and pet comes to you and does nothing, which sounds like 5th. But oh well.
/petpassive [target=mouseover, noexists] (again, doens't matter if you check the color the the targets shoulder armor, result is pet comes to you if no mouseover)

Again would use keybind for petpassive, but you have to remember to turn it back off in your attack macro. Just use /petdefensive to make pet normal. Also, 5 and 6 are the same (unless I am missing something). It doesn't matter if you check for the stars in the sky, no mouseover, petpassive.
/petpassive [target=mouseover, noexists]

If I am not understanding what you are asking, I am sorry. I just try to boil things down the basic items. I also have a petattack and petfollow keys (I use F1 and F2) since I use a lot of pet classes. But they are the same as CTRL-1 and CTRL-2, just easier for my fingers and muscle memory at F1, F2. I also put /petattack [target=focus-target] on my alts so the pets always attack what my main is attacking, no matter what they were attacking before. No target on my main, no attack. Yeah, the pets will still finish off whatever they had targeted, but I just push the follow key (CTRL-2) or just run out of range (BG's).

Hope that helps and good luck.