It will only cast renew because square brackets are conditionals. By putting /cast [target=x] heal first, you're locking out everything else. /cast [target=x] heal will always succeed, and in doing so, knock out your GCD. What you want to do is put the ones that will fail first, like this:

/cast [target=x,modifier:shift] heal
/cast [target=x] renew

That way, once the [modifier:shift] conditional fails (if you're not holding shift), it will fall through to /cast [target=x] renew, which will always succeed.

As a side note, this fall-through only works with conditionals in []'s. If a spell itself fails, the whole macro fails (since a recent patch). So for instance:

/cast [target=x] judgement
/cast [target=x] holy light

will fail whenever judgement's cooldown is not up, even though the holy light part might have worked.