I read on arenajunkies.com that if you your totems will work if you wait for the opposing team to show up.

I made this change and this is my favorite arena now. Traded a Flame Shock on one guy for a frost shake and they rarely start with 5 people.