It will be a huge hit to your DPS output. The biggest difference you will see is that anytime you enter an instance with more than 2 toons, the instance creates additional mobs (usually strong or elite). The difficulty of the adds are designed for 4 players so anytime you go in with 3 you are at a disadvantage. Now, this doesn't happen for every instance so might be able to get away with having 3 sometimes.