Having 5 tailors is not wasted, you still get the benefit of 4 extra tailor cooldowns. You're not missing out on any extra cloth either, you're just not getting 5 times as much.