I have two internet lines in my house. My main is a Covad DSL line running at 1.5mb and my second is a Comcast line running at 16mb. The covad line runs sweet, usually 150ms (my guilds wow server is on the other side of the country). My comcast line however has horrible latency during certain parts of the day and gives me the same issues as you; 200-300 latency and sometimes one or two of my toons will shoot up real high like 500 to 2000.

Could just be your isp