Yep, I agree with both of them ^^. Right now, 99% of my armor is cloth with some leather. I wouldn't worry about it at low levels, as for me, I'm level 51 (almost 52) and my lack of mail hasn't held me back a bit.

Good luck!