I'm sure having only four is doable, but you might run into some issues on certain instances later on. You also can't use the LFD tool unless you want a random person to join in with you and you'll be stuck walking to dungeons yourself. You could always be a total dick and just kick the random person, but... that's kind of being a dick. >_>

As for running five, you could always just use your A or B account to recruit another. It'd be a confusing RaF recruitment tree, but as long as everyone was running together there would be no issues.