In order for them to look exactly right, the physical size of the video feed needs to be the same in both windows.

By that I mean, if you take a screenshot with both windows showing -- one being large and one being 1/4 of the second screen as per your Window Layout -- open it up in photoshop or something and compare the size of your view on the main screen, to the source on the smaller screen.

If they don't exactly match, then it's either going to scale up or down from that resolution (in the screenshot, not the main window size when the window swaps back to it) to provide your feed. If it has to scale up (because the view bigger than the source is at that time) then it's going to look pixellated.

Which is why Window Layouts are beginning to change, and the Stacked full screen style from the wizard is optimal for video feed usage.