Need Help with Hacked VPS

**Frank Broughton** · 11-30-2007, 07:28 PM

Recently had these files uploaded to my VPS (CentOS WHM/cPanel):

Code:

/tmp/.ICE-unix/up

-bash-2.05b# ls -l

total 10080700

-rw-r--r--    1 nobody   nobody   713555968 Nov 23 18:25 1408.2007.iTALiAN.MD.DVDRip.XviD-SVD.avi
-rw-r--r--    1 nobody   nobody   733859840 Nov  4 06:49 2061.Un.Anno.Eccezionale.2007.iTALiAN.MD.CAM-P3V4.avi
-rw-r--r--    1 nobody   nobody   741101568 Nov  7 09:39 C_era.Una.Volta.In.America.(Robert.De.Niro).1984.iTALiAN.DvDRiP-NwP.cd1.avi
-rw-r--r--    1 nobody   nobody   741175296 Nov  7 09:42 C_era.Una.Volta.In.America.(Robert.De.Niro).1984.iTALiAN.DvDRiP-NwP.cd2.avi
-rw-r--r--    1 nobody   nobody   720089088 Nov 18 17:51 I.Fratelli.Solomon.2007.iTALiAN.LD.DVDSCR.XviD-SiLENT.tar
-rw-r--r--    1 nobody   nobody   703627264 Nov 21 06:06 Il.Risveglio.Delle.Tenebre.2007.iTALiAN.LD.TS.XviD-SiLENT.tar
-rw-r--r--    1 nobody   nobody   703854592 Nov 17 13:43 La.Leggenda.Di.Beowulf.2007.iTALiAN.MD.TS.XviD-SVD.avi
-rw-r--r--    1 nobody   nobody   891047936 Nov 23 12:50 Piano.Solo.2006.italian.dvdrip.Drammatico.SilverGroOve.avi
-rw-r--r--    1 nobody   nobody   731187892 Nov 21 17:38 Rapimento.E.Riscatto.2000.italian.dvdrip.ac3.cd2.Drammatico.SilverGroOve.avi
-rw-r--r--    1 nobody   nobody   727746768 Nov 16 19:04 SmS.Sotto.mentite.spoglie.2007.iTALiAN.REPACK.MD.TS.XviD-NuCooL.avi
-rw-r--r--    1 nobody   nobody          0 Nov 29 17:05 Sono.Cose.Che.Capitano.Ficarra.E.Picone.2007.iTALiAN.DvDRip.DivX-PdB.tar
-rw-r--r--    1 nobody   nobody   729124864 Nov 29 20:56 The.History.Boys.2006.iTALiAN.DVDRip.XviD-SVD.avi
-rw-r--r--    1 nobody   nobody   721565696 Nov 17 12:29 The.Matador.2005.iTALiAN.LD.DVDRip.XviD-SiLENT.tar
-rw-r--r--    1 nobody   nobody   733505536 Nov 29 17:19 Vasco.Olimpico.07.2007.iTALiAN.Ac3.DvDRip.DivX-PdB.cd2.avi
-rw-r--r--    1 nobody   nobody   721035264 Nov 18 07:32 Waitress.Ricette.D.Amore.2007.iTALiAN.MD.DVDRiP.XviD-GOLD.tar

Got a nice $200 bandwidth overage bill for the pleasure of letting some scumbuckets download 800 GB of these files.

I have checked to make sure I have removed scripts I am not using and have updated versions of PHPBBS - PHPList & Wordpress left installed.

What logs would I start to investigate to see how these files got uploaded?

Thanks for any help!

-=Frank=-

**elflaco** · 11-30-2007, 08:55 PM

Someone else will have to answer the questions about where to look (have you opened a ticket with support by chance?), but I am curious if you took certain hardening steps like changing the SSH and FTP ports, using 16 character passwords and installing APF and BFD or CSF to alert you about access attempts and ban the perpetrators?

Not trying to be condescending, just trying to help you narrow things down, cause that's a buttload of stuff for someone to just throw up there with you unawares.

If you did follow those steps I would guess either a naughty script or unpatched vulnerability...

**Connie** · 11-30-2007, 10:10 PM

I thought if you exceeded bandwidth you account was shut down. Apparently I'm wrong. That would be one h*** of a surprise bill.

Oh well your church can afford it.

**Vin DSL** · 12-01-2007, 12:14 AM

Originally Posted by Frank Broughton

What logs would I start to investigate to see how these files got uploaded?

Um...

How many logs do you have?

They probably used Wget, but I'm just guessing...

You got a photo gallery, or whatever, that allows ppl to upload things? If so, that would be suspect too.

I dunno, without checking your log...

Hrm...

"What logs..." mystifies me!

What do you mean by that?

**thisisit3** · 12-01-2007, 02:59 AM

The /tmp/.ICE-unix directory is used for X sessions. In this case, someone used it to store those files and then used some way to share them to others.

The questions you should ask yourself are:

1) If he stored those files there, it probably means he doesn't have root access, but you can't be sure, did you scan the entire VPS for suspicious files? Did you install rkhunder to search for rootkits?

(if you even suspect that he got root access then you need to ask support to re-install your VPS from the start!)

2) If he didn't get root, then which account was vulnerable? and under that account, which script or web application?

You need to look at the HTTP and other logs, to see when the attacks started, how they got in etc.

**thecoalman** · 12-01-2007, 05:36 AM

Originally Posted by Connie

I thought if you exceeded bandwidth you account was shut down. Apparently I'm wrong. That would be one h*** of a surprise bill.

I'd rather they start charging myself, a notification that you have gone over might be appropriate so at least you are aware of it so you don't get hammered with a big unexpected bill. I check my stats at least every couple of days so I'd probably catch something like this myself.

**JPC-Masood** · 12-01-2007, 08:04 AM

All those files and folders are owned by user nobody, indicates that this was some web script exploit. No root level exploit when I did the audit of this vps. If you follow this; http://www.jaguarpc.com/support/kbase/731.html you will be able to track it down.

**Frank Broughton** · 12-01-2007, 10:15 AM

Originally Posted by Vin DSL

Um...
How many logs do you have?
"What logs..." mystifies me!
What do you mean by that?

/usr/local/apache/logs
access_log
error_log
sslengine_log
suexec_log

/usr/logs/apache/domlogs
ftp and web logs for 21 sites

There are MTA (exim) logs also. Some dom logs are huge as I have not purged them in a long time. I am downloading the above logs now, are there others I need to check?

Thanks for the help. I am no security expert obviously. I can get apps to run and read directions, but trouble shooting errors is above me.

Maybe I should switch my sites to a combimnation of sdx and reseller as most are done for friends and I make no money off this.

@Connie.... At least I am the treasurer of the church... so I had no fight in getting it paid - hey wait JPC just charged my cc with no notice of invoice. That teed me off somewhat too.

@Masood.... I am a following that KB thanks, but I am no expert in this, did you check the note I added to the ticket about making sure my site is susupended when bandwidth reaches 500GB in a month. I DO NOT WANT TO WASTE $200 again.

Maybe I should just upload files to fill my VPS to capacity and they would have no room to upload anything.... that would work no? Only have 17.5GB to fill...

-=Frank=-

**Vin DSL** · 12-01-2007, 10:52 AM

Originally Posted by Frank Broughton

/usr/local/apache/logs
access_log
error_log
sslengine_log
suexec_log

/usr/logs/apache/domlogs
ftp and web logs for 21 sites

There are MTA (exim) logs also. Some dom logs are huge as I have not purged them in a long time. I am downloading the above logs now...

I see...

Well, I guess it all depends on HOW you think they got in, and what you THINK they did - but this is all a mystery at this point... soooo...

I'd do what you're doing... download them all, then use Windows Grep Search Utility to search for one of the file names.

This is tedious work, so be patient!

**Vin DSL** · 12-01-2007, 10:57 AM

Originally Posted by JPC-Masood

All those files and folders are owned by user nobody, indicates that this was some web script exploit...

Excellent point!

I noticed that last night, but was too tired to comment on it. Plus...

I wasn't sure if VPS was handling file ownership the old way or the new way...

Just curious - if someone uses the cPanel file manager to upload files, does nobody (still) own it, or has that changed too?

Hrm...

**thisisit3** · 12-01-2007, 11:00 AM

he is on a VPS, so why download anything or try to use crappy windoze, when all he needs is to ssh to his VPS and "grep" things around...

**Frank Broughton** · 12-01-2007, 11:15 AM

Vin.... Me thinks I have no idea how they got in. If I could find the suckers I would forget who I am right now and become what I was and smash their head into the curb a few times....

Patience... thanks I needed the reminder. This has been one long four weeks for me. First, I total my truck, then my handicapped son goes into the hospital for two weeks with pneumonia, then I find out my cousin is going to jail for four years. Then this, a $200 bill for some stupid hackers.

I have always been confused as to what logs are for what, I have researched with google to confusion on my part and was always embarrassed to ask questions here as to what does what. I have played with computers for a long time and sometimes the simple things are the things I lack knowledge in for most of my learning has been through trial and error. I know some complex things well yet some simple things confuse me. I hope I am not alone in this.

Having dial up at home does not help much when playing with these log files. I am at the office right now downloading the logs with DSL.

A nice tutorial on log usage as to JPC specific VPS's would be nice. Seems the same logs are in different directories on the VPS causing me more confusion...

I think I will just go visit my dad's grave site and cry.....

**Vin DSL** · 12-01-2007, 11:16 AM

First things first...

Originally Posted by thisisit3

he is on a VPS, so why download anything or try to use crappy windoze, when all he needs is to ssh to his VPS and "grep" things around...

Originally Posted by Frank Broughton

Thanks for the help. I am no security expert obviously. I can get apps to run and read directions, but trouble shooting errors is above me.

Secondly, you might find this interesting, Frank...

Any ideas about this hack...?

**Vin DSL** · 12-01-2007, 11:26 AM

Originally Posted by Frank Broughton

I think I will just go visit my dad's grave site and cry.....

Bah!

These are ALL blessings, Frank!

"Blessed is the man whose strength is in you, in whose heart are the ways of them, who passing through the Valley of Baca, make it a well; the rain also fills the pools. They go from strength to strength—every one of them in Zion appears before God." Psalm 84:5-7

It might not look that way, right now, but all this stuff will make you stronger!

The trick is to get out of the 'Valley of Weeping' as quickly as possible! The problem is, there WILL be another valley.

Sitting in that valley and crying won't do jack sh!t...

**Daniel_DBS** · 12-01-2007, 11:50 AM

Originally Posted by Frank Broughton

I have always been confused as to what logs are for what, I have researched with google to confusion on my part and was always embarrassed to ask questions here as to what does what. I have played with computers for a long time and sometimes the simple things are the things I lack knowledge in for most of my learning has been through trial and error. I know some complex things well yet some simple things confuse me. I hope I am not alone in this.

Your not alone Frank! I am the same way, the simple things always confuse the heck out of me and then the harder stuff is a breeze, and like you, I am not exactly sure what log is for what, I just followed most of the tutorials I have found on the web about securing my VPS and that is what I have done, BFD, root kit checker, etc...

I hope you figure this one out!

-Daniel

LinkBack

Thread Tools

Display

Need Help with Hacked VPS

Bookmarks

Bookmarks

Posting Permissions