I haven't tried it, but I remember reading somewhere... ah, here it is! According to this thread - http://www.dual-boxing.com/forums2/viewtopic.php?t=1767 - it is possible to set the resolution in the file /Applications/World of Warcraft/WTF/Config.wtf.

Look for the line that says SET gxResolution "1024x640" and try to set it lower than 800x600. Hope it helps!