You actually have a very limited time to hit him with harpoons after the "in range" announcement.

I find that I usually only have time to shoot off 2 harpoons (including movement) before they stop registering.

I'd say that based on you having fired 6 harpoons in one flyby, most of them probably fell outside of the effective window.