My best is 10 gems, on 5 transmutes (one group doing transmutes).
It was a 4 gems, 3 gems, 1 gem, 1 gem, 1 gem.

I've yet to see 5 gems, on a single transmute.