Site issues 6/13 - Update / Explanation

So Monday was a pretty crazy day around here. It started off with the site being completely unresponsive / down. Turns out the host had turned off the VM because there was a script running that was eating up all the server resources. They sent us an email with the script details. It was not a script I was familiar with. They allowed our VM to come back up and gave us 2 hours to "Fix the script, or be shutdown again" I wondered how we were supposed to fix the issue without the VM being up.

When the VM came back up I immediately began checking our data for signs of corruption, breach, or removal. I looked at traffic logs and system logs and found no trace of our databases being touched. All of our backups were intact and MD5s all checked out. I ran Rootkit Hunter and chkrootkit. Chkrootkit came back with a clean report. Rootkit Hunter found 2 possible rootkits. I began digging to see if either of the possibilities were active and they were not. This could have been just a false positive, I did not have enough time / tools to properly determine. We went back to the host and showed the logs of both scans and asked them to confirm infection. We also asked for a new VM to move our files to. They refused to issue us a new VM based on the fact that having a root kit was "our fault, and we should fix it". They did let us know they would issue a ticket to their exploit department to check out our VM.

Astounded at the terrible help the host was being, I began to attempt to remove the rootkits. I pulled off some of the recent backups and I tared up the sites and pulled them all down and scanned it all with AV software. Once I was comfortable I had clean backups I started hacking at the VM. I was following directions for the removal of the first rootkit when I needed to reinstall some of the base applications. I reinstalled the base applications, and then patched up the OS with security fixes. Here is where things went completely sideways. One of the security patches was to bring up MySQL to a new version. Apparently that version was incompatible with the Virtuozzo VM software and the VM refused to allow MySQL to restart. I instantly reverted MySQL to the previous version and it was still unable to start. We contacted the host to see if they could offer some insight (or you know, FIX IT). They replied with "you don't upgrade the OS of a VM". I immediately replied "What kind of bullshit are you running here? I upgrade Xen, OpenVZ, VMWare, and VirtualBox VMs on a near daily basis!?" Their reply was: "You can upgrade the OS on our Xen VMs, but you didn't ask for Xen, we gave you the basic VPS VM". So I then threw our forum database to a remote MySQL server (that I just happened to have) and the site was back up, although a bit slow.

The host's response to our request to fix MySQL was to blank the VM. Which nuked ALL of our data. Luckily enough I had already snagged the site files, database backups, and the raw MySQL data files. So in a twisted manner, we got what we needed, a clean VM. I never did find any sign of the original script they shut us down for. I really question that as well. It was a perl script, and nothing we run here uses perl. I don't even use perl here at the system level.

TL;DR:
Our data is safe, our host is terrible.

You, sir, are pretty goddamned awesome.

Shame the host sent you on a snipe hunt. Seems most good sites run despite the help of the host and only because of the sheer determination of an awesome admin.

/salute

- Souca -

Possibly moment of clarity... I may actually be able to post

Thanks for the kind words. Keeps me going during times like these :_D

Thanks for keeping us up; I really like the community here and would hate for it to go away. Cheers to a job well done.

Thanks, I appreciate all the effort you are putting into the site Svper.
But I'm not going to congratulate you on a job well done til I can confer with my GM friend on this matter.

*gives Svper a pizza and a case of beer*

It's about time shit changed around here, now I can post all these guides I've been teasing people with.

There were still lingering issues as you may have seen today. Certain settings the host didn't put back that caused the site to seem down, it was just more traffic than their average user and they needed to modify.

It's somewhat up stable now but with Svper handling it I'm looking into a new host that will give us twice as much resources for the same price.

Thanks Svper for being awesome.

Do you take donations? I wouldn't mind shelling out few dollars if that means we get a better host :)