5x > 4x

You can do anything you do 4x with 5x. You just leave one out. One will not "fall behind" because you just need the one furthest ahead out and it will even out over time.

i.e.
If you only want bring 4 to an instance pre-70, leave out the one with the highest xp.
If you only want to bring 4 to an instance at 70, leave out the one with the highest rep in that faction.
If you only want to bring 4 to a BG, leave out the one with the highest honor.
If you only want to bring 4 to arena, leave out the one furthest from the median PR.