Last night I noted that my internet connection slowed to a crawl. I popped into my server and checked a few things (mainly top and netstat), and noted a number of connections on port 6667, and a couple of programs running (particularly 'eggdrop'). After some searching, program removal, root password change, addition to my firewall ruleset to block the IP range which maintained the irc connection, and upgrade of some potential holes (postfix, apache, others), I hope this is over. I still don't know how for certain they got in, and I'm sure I'll see more of this activity in the near future (it occured 4 months ago on a different distro/install), so I plan to be vigilant, but I am really irritated that this occured (and a little concerned, as I am barely a novice when it comes to Linux security). I keep my software up-to-date, but it seems that open eyes are all you really can combat this sort of thing with. Anyone have any stories to tell?

You should probably reinstall your distro, a common feature of the click-and-play rootkits that a lot of skiddie fucks use (you can tell it's a skiddie because it's about as subtle as a brick to the face) are subverted binaries -- even your /bin/ls might have a backdoor in it.

Then install tripwire so you can be sure of this in the future. And consider moving to OpenBSD, which is relatively easier to set and forget.

Also turn off any nonessential services - for example, if you have something like fingerd running and you don't absolutely need it, turn it off.

One thing that would be smart to do is to make sure there's a firewall between the server and your LAN preventing any further access to your network.

If you can afford it, you might want to consider not running a server yourself and let someone who has more experience with the network security side of things take care of it. However, if you really want to run Apache yourself look into putting it into a chroot jail. More info here.

Also as Ravuya pointed out a clean reinstall is probably a good idea - you can't trust any of the apps on your system. You're lucky that the moron that did this wasn't smart enough to also replace your top and netstat with his own custom versions that hid eggdrop and the port 6667 connections from you.

You should also create a non-privileged user for each of your networked daemons to run as. Running apache as root is asking for trouble.

To avoid this in the future.

Check the server for root kits:

http://www.rootkit.nl/projects/rootkit_hunter.html
Small program, excellent. Run this periodically and have it e-mail you the results.
Also, be aware the top and netstat and ps and many other binaries may become modified by some rootkits, so you can't always trust the results of your snooping.

Tarball up your /usr/bin and /bin directories, in case, forbid you get rootkitted...
You can use a real version of the binary to snoop around.

Secure your /tmp and /var/tmp directories. Many, many rootkits, scripts and exploits rely on the fact that you may have execute permissions for nobody in those directories.

http://www.linuxtechs.net/securing_tmp_partition

Do a TCP dump if you think you've been compromised.. Look for anything suspect..

tcpdump > /tmp/tcpdumpout-txt.

Only leave open, sourcing from your server, what is ABSOLUTELY essential. Likewise for inbound traffic. Run ssh on a random port, if you use ssh. Never run it on 22, so many brute force ssh attacks in the last year.

Check your /etc/.shadow file for UID's with a 0.. (root enabled logins)

As a general rule of thumb, anything with PHP and Apache is suspect to exploits. Try to avoid PHP if you can..

And lastly, if you get ambitious, look into backups. Bare metal restores are easier than you (and a lot of others) might think. If you use mysql, look into the mysqldump binary to back up all your databse files. O'Reilly has a good book on Unix DR which outlines this stuff. If you can't afford to lose it, back it up. If you have the space on a networked drive, or a sepearte partition that's grand. Look into making nightly tarballs, excluding the /proc, /mnt/anythingexternal, /tmp and so on.. That with a backed up, fdisk -l output and your /etc/fstab is almost everything you need for a bare metal restore.


Hope this helps.