Speed does not equate to latency. When they throttle you for excessive bandwidth use, it just limits your transfer rate, not how long it takes a packet to get to the server and back.

I'd suspect your ISP has a sucky network in your area that gets overloaded during prime time. Your router should not be doing that unless you have an open WiFi setup and someone else is using yours during those hours.