The deserter buff would only occur if you leave the group you are in after leaving the dungeon. So as long as you stay in the group you are good to go.
It is not aimed at forcing people to do dungeons they do not want to do, rather to discourage people leaving a group at the beginnning of the dungeon when everyone else still remains to do the dungeon.