Unfortunately, it seems like Rift is a bit labile in terms of performance. One day, I get 55-60fps/30fps in a certain area and the next I get 14/12. If you are consistently seeing this drop when adding group members, I'll try to duplicate it for you (right now everyone joins at once heh).