You can do a very simple test to show that deaths do not subtract 1 from your instances per hour limit. Runn more than 1 char into a new instance get one naked, die, and run back 5 times, then run out with both reset and run back in, you'll see no problem with either one getting in. I believe your time insude the instance is counted and the time the toon spends outside is not counted down from your instance time in an hour.

So say you have 1 toon inside for 1 hour 2 min (enough to make sure he is over the limit) and the other toon dies twice and spends 52min inside and 10min outside in the hour. The second toon will get the message about too many instances in an hour the first will be fine to continue with another instance.