I tested my addons one at a time last night.

Turning off Questhelper gave me back 10-15 FPS and reduced the strain on my processor.
Turning off Recount got me about ~3 FPS

Please note my computers suck, so this was a huge improvement for me. :P

Mods that had no change in performance:
Xperl
Jamba
SmartBuff
Omen
Macaroon
Atlas Loot
Tomtom
Liteheaded