This happens to me whenever I use the mouse to change my view within a smaller instance of wow. It doesn't happen with larger instances of wow.

If you change your view too much, then it seems like it'll bounce out and you won't be able to move our mouse into that instance until you switch that window to your main window and right click or fix your view.

My guess is it reacts like the mouse is moving outside that instance of wow. To understand the effect, try holding your forward movement key and moving across the windows. Your characters will now continue to move forward until you cross the window again without holding a button. Point being - small mouse view changes are probably ok, large views will work, but large changes in small views are buggy.