Depends on which GPU you are looking at moving too. Most of the newer ones will have more vRAM.
Also depends a little on how many pixels you want to drive (physical size of the screen doesn't matter), and for games, at what texture quality/AA settings.

Easiest method would be to set your graphics to the quality you want, and then monitor vRAM usage, to see how much it needs. Extrapolate that to 5 clients. Not that it will necessarily be a true measure, but if you start 1, then 2, then 3, with the graphics settings you want, you should be able to get a reasonable estimate of the desired vRAM for 5.
As a general indicator, the first 4-600MB will be used by Windows.