You Had a Bad Day
Comments(6)
So I just discovered that my dedicated machine, which is generally doing absolutely nothing, was running at a load average of about 1. The top CPU abuser? Some command I didn’t recognize (barbut). I was immediately suspicious. I killed the process, then noticed that it had been running as the cvs user, so I ran a ps to find all commands running as cvs.
webkill?
Yes, that’s right, my dedicated box was an involuntary participant in a distributed denial of service attack, orchestrated by an IRC bot, also known as barbut (which I found, source and all, in /home/cvs/).
Time for damage control. First, I obliterated the user cvs. Then I installed and ran rkhunter; the “good” news is that no root kits were found. Then I went to change the SSH port — oh, wait, I’d already done that, but never restarted with the new config: shame on me!
One of the unfortunate side effects of using CVS over SSH is that you need accounts with shell access. Apparently I’d created a user with a basic password to allow friends to check code out of my local CVS server; I’m guessing that password just got brute-forced. There doesn’t look to be anything else amiss, so I guess I was somewhat lucky.
Anyone want the source code to an IRC bot?
Passwords are so often the weak link. No matter how good the door if the lock is no good there is a problem. Glad you found it and that nothing worse was done.
You should get rid of the dedicated machine! 😉
Can you check “last” to see where they logged in from? Although, in all probability, it’s another infected machine.
You can also check your bandwidth usage ( https://encompass.layeredtech.com/ ) — the good news is that on a 10 Mbps, it takes a couple weeks to hit 1TB.
If you run ssh on port 22 and watch your log files, these days you’ll see thousands of attempts a day. Literally, they just sit there and try common usernames and passwords. I use DenyHosts, which will ban anyone who fails at logging in 5 times in a row. I’m pretty strict because I never fail more than once (on a typo), and because I can get in from multiple IPs easily. You could set the limit at 50.
Heh, glad you posted this! Writing that just reminded me that I haven’t gotten e-mails on banned IPs in a while… Turns out that DenyHosts wasn’t set up to start at boot, so when the box died a few weeks back, it never restarted…
Mar 26 14:27:12 oxygen sshd[24986]: Invalid user router from 190.144.140.83
Mar 26 14:32:34 oxygen sshd[28543]: Invalid user mambo from 190.144.140.83
Mar 26 14:32:35 oxygen sshd[28554]: Invalid user rusticos from 190.144.140.83
Mar 26 14:32:36 oxygen sshd[28565]: Invalid user rusticos from 190.144.140.83
Mar 26 14:32:37 oxygen sshd[28576]: Invalid user test from 190.144.140.83
Mar 26 14:32:38 oxygen sshd[28587]: Invalid user test from 190.144.140.83
Mar 26 14:32:39 oxygen sshd[28599]: Invalid user test from 190.144.140.83
Mar 26 14:33:05 oxygen sshd[28895]: Invalid user toor from 190.144.140.83
Mar 26 14:33:06 oxygen sshd[28906]: Invalid user toor from 190.144.140.83
Mar 26 14:33:07 oxygen sshd[28918]: Invalid user toor from 190.144.140.83
Mar 26 14:33:08 oxygen sshd[28929]: Invalid user administrator from 190.144.140.83
Mar 26 14:33:09 oxygen sshd[28940]: Invalid user dasusr1 from 190.144.140.83
Mar 26 14:33:10 oxygen sshd[28951]: Invalid user dasusr from 190.144.140.83
Mar 26 14:33:11 oxygen sshd[28956]: Invalid user dasusr1 from 190.144.140.83
Mar 26 14:33:12 oxygen sshd[28967]: Invalid user dasusr1 from 190.144.140.83
Mar 26 14:33:13 oxygen sshd[28978]: Invalid user dasusr1 from 190.144.140.83
Mar 26 14:33:14 oxygen sshd[28989]: Invalid user dasusr1 from 190.144.140.83
Of course, that’s not all…
oxygen ~ # grep 190.144.140.83 /var/log/messages | wc -l
19945
Anyone want the source code to an IRC bot?
Source code, not so much. Server / user / password, sure! Years ago when my web-facing desktop got “hacked,” it joined an IRC machine, so after killing it, I joined and just watched.
Edit: pushing my own comment through moderation.
They must not have used the bot much: when I went to reboot the box (accidentally shutdown all ssh connections and the daemon), I looked at my bandwidth usage; there was barely any.
I’d seen tons of attempted logins in the past, but was never real worried because they were trying accounts that I’d never have….
We had a similar bot appear on one of the servers at work, and we eventually hacked the source into a daemon that would just log all the activity in the channel. But eventually that got boring. :-p
I tracked down the IP, and it does appear to be another compromised host. Unfortunately, I couldn’t find an email address to warn them.
The attacker left a lot of info behind, if it seems interesting, maybe I’ll post some of it.
I guess I never mentioned: the initial login attempt occurred on March 11th, about 2 weeks ago. But, like I said, they didn’t use much bandwidth.
Unfortunately, I couldn’t find an email address to warn them.
You could always whois the IP and contact that host’s abuse department… I always have a hard time deciding whether doing things like that is nice and helpful, or if I’m coming across as a jerk who’s getting someone’s machine suspended.
Who breaches security to launch attacks and then… doesn’t launch attacks?
[…] Bandwidth graph of the period during which my machine was compromised: […]