Site issues 6/13 - Update / Explanation
So Monday was a pretty crazy day around here. It started off with the site being completely unresponsive / down. Turns out the host had turned off the VM because there was a script running that was eating up all the server resources. They sent us an email with the script details. It was not a script I was familiar with. They allowed our VM to come back up and gave us 2 hours to "Fix the script, or be shutdown again" I wondered how we were supposed to fix the issue without the VM being up.
When the VM came back up I immediately began checking our data for signs of corruption, breach, or removal. I looked at traffic logs and system logs and found no trace of our databases being touched. All of our backups were intact and MD5s all checked out. I ran Rootkit Hunter and chkrootkit. Chkrootkit came back with a clean report. Rootkit Hunter found 2 possible rootkits. I began digging to see if either of the possibilities were active and they were not. This could have been just a false positive, I did not have enough time / tools to properly determine. We went back to the host and showed the logs of both scans and asked them to confirm infection. We also asked for a new VM to move our files to. They refused to issue us a new VM based on the fact that having a root kit was "our fault, and we should fix it". They did let us know they would issue a ticket to their exploit department to check out our VM.
Astounded at the terrible help the host was being, I began to attempt to remove the rootkits. I pulled off some of the recent backups and I tared up the sites and pulled them all down and scanned it all with AV software. Once I was comfortable I had clean backups I started hacking at the VM. I was following directions for the removal of the first rootkit when I needed to reinstall some of the base applications. I reinstalled the base applications, and then patched up the OS with security fixes. Here is where things went completely sideways. One of the security patches was to bring up MySQL to a new version. Apparently that version was incompatible with the Virtuozzo VM software and the VM refused to allow MySQL to restart. I instantly reverted MySQL to the previous version and it was still unable to start. We contacted the host to see if they could offer some insight (or you know, FIX IT). They replied with "you don't upgrade the OS of a VM". I immediately replied "What kind of bullshit are you running here? I upgrade Xen, OpenVZ, VMWare, and VirtualBox VMs on a near daily basis!?" Their reply was: "You can upgrade the OS on our Xen VMs, but you didn't ask for Xen, we gave you the basic VPS VM". So I then threw our forum database to a remote MySQL server (that I just happened to have) and the site was back up, although a bit slow.
The host's response to our request to fix MySQL was to blank the VM. Which nuked ALL of our data. Luckily enough I had already snagged the site files, database backups, and the raw MySQL data files. So in a twisted manner, we got what we needed, a clean VM. I never did find any sign of the original script they shut us down for. I really question that as well. It was a perl script, and nothing we run here uses perl. I don't even use perl here at the system level.
TL;DR:
Our data is safe, our host is terrible.
Reply With Quote
Originally Posted by Khatovar
Connect With Us