Conficker

People are already calling Conficker the worst worm in years, even though bloggers in Europe (where it’s already April 1) are saying it’s not doing anything.

It looks like ESet, makers of the fine NOD32 anti-virus software, have a page on how to remove Conficker, including what seems to be a free tool for its removal. Based on my limited reading, it seems that installing the latest security patches for Windows pretty much render your immune. But we all know people who don’t do that.

It looks like Conficker alters some Windows network internals, causing it to exhibit some different fingerprint characteristics when probed, so tools like Nessus and nmap are apparently able to detect it. Though Nessus is a gigantic problem, and I don’t have any machines with Conficker on my home LAN, so I can’t confirm this.

E-mail Holding Tanks

I suspect my situation really isn’t that different from a lot of people. I get a lot of e-mail every day. Easily 250 messages on a good day, and well over 1,000 on a bad day. I get a copy of every error that occurs on our site, an e-mail for every cron job that runs anywhere, and a whole lot of other junk. I have Exchange set up to filter my mail before it lands in my Inbox, which helps keep me sane.

Some mail I want to keep forever. I consult my “Sent Items” folder often enough that I don’t intend to ever purge it. Same goes for my “Archival – Keep” folder. But I also have a lot of folders on the mailserver that are basically holding tanks. Periodically I go through and clean out the accumulated crap. I realize I don’t need copies of errors that occured in January, or cron jobs from December, and go crazy deleting things.

There are a few problems with that. The biggest, really, is that it’s a waste of my time. But I’m sitting here blogging about it, so I can’t complain about that. 😉

The other one is that Exchange (and probably any other mailserver) really, really hates it when you highlight 10,000 messages and hit delete. And by “Exchange really, really hates it,” I mean you start hearing everyone in the office asking each other, “Is e-mail down for you, too?” So I learned my lesson and wrote nice-delete.pl, a simple little Perl script using Net::IMAP to grab a list of all mail in a folder and iterate over the messages, deleting 50, expunging the mailbox, and then sleeping for 3 seconds. That keeps the load down.

I’ve thought about extending it to run weekly via cron, download each message, and check the headers to see if it’s older than, say, 21 days, and just trash it if so. But then I realized: I use Thunderbird, which has an ungodly amount of plugins. Surely, someone has written this before!

Unless my Google-fu is failing me, though, no one has. Is it really that uncommon of a request to want to auto-purge certain folders after a while? And, more importantly, what does it take to write a Thunderbird extension?

Viewing all cron jobs

Periodically I run into the situation where I’m trying to find a cron job on a particular machine, but I can’t remember which user owns it. At least on CentOS, it’s easy:

cat /var/spool/cron/* will show all crons. The crontab command doesn’t seem to support doing this. The downside is that that command just mashes them all into one list, which is only useful if you don’t care who the job runs as. Usually I do. Here’s a simple little script to format the output a little bit:

for i in `ls /var/spool/cron/`; do
        echo "Viewing crons for $i"
        echo "--------------------------------------"
        cat /var/spool/cron/$i
        echo
done

SHOW CREATE TABLE

Silly MySQL trick of the day. This is probably no stranger to those work with MySQL all the time, but it’s a nice trick to have up your sleeve.

For reasons I still can’t wrap my head around, one of the MySQL tables we need for Cacti just up and vanished. It’s not really used for anything essential, so it’s not a great loss, just really annoying. So I need to recreate it. Fortunately, I have MySQL running on another machine.

mysql> desc poller_output;
+---------------+-----------------------+------+-----+---------------------+-------+
| Field         | Type                  | Null | Key | Default             | Extra |
+---------------+-----------------------+------+-----+---------------------+-------+
| local_data_id | mediumint(8) unsigned | NO   | PRI | 0                   |       |
| rrd_name      | varchar(19)           | NO   | PRI | NULL                |       |
| time          | datetime              | NO   | PRI | 0000-00-00 00:00:00 |       |
| output        | text                  | NO   |     | NULL                |       |
+---------------+-----------------------+------+-----+---------------------+-------+
4 rows in set (0.00 sec)

It’s not that complicated. I could take a second and write out the CREATE TABLE statement I’d need to run on the machine missing the table. But here’s a neat trick (run on the system with the table, obviously):

mysql> show create table poller_outputG
*************************** 1. row ***************************
       Table: poller_output
Create Table: CREATE TABLE `poller_output` (
  `local_data_id` mediumint(8) unsigned NOT NULL default '0',
  `rrd_name` varchar(19) NOT NULL default '',
  `time` datetime NOT NULL default '0000-00-00 00:00:00',
  `output` text NOT NULL,
  PRIMARY KEY  (`local_data_id`,`rrd_name`,`time`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

Why write out the CREATE TABLE statement when we can just let MySQL tell us exactly how to re-create a table? I initially discovered this with the considerably more complex issue of views (SHOW CREATE VIEW), but it extends to tables, too. So now we can just copy-and-paste. Viola!

Well not viola for me, but viola for anyone whose filesystem hasn’t gone corrupt.

Handling Swap

Here’s a neat tip I picked up from Andrew. Have you ever run into a machine that won’t reclaim all of its swap? It turns out there’s an easy way to force a reclaim.

First, a caveat. For most people, this makes no sense. If something is just sitting in swap even when there’s lots of free RAM, it means that whatever’s in swap is something that nothing, anywhere, cares about. It would have been reclaimed if anything tried to use it. You want to leave it in swap. And a second caveat: what I’m about to say is an awful idea if you don’t have lots of free RAM. But in cases where you’re using 4% swap and getting paged every 15 minutes because the monitoring system sees swap being used, and yet you have 5GB of free RAM, here’s a neat trick that will force everything in memory to be reclaimed.

First, find the swap partition. On Linux, it’s “cat /proc/swaps,” though you could also look at fstab. GNOME and KDE users probably have a fancy GUI for this, but our servers don’t run GNOME. 😉 On the machine I just did this on, it was /dev/sda2, so I’ll use that as an example.

Then, you ‘unmount’ the swap partition, and then re-enable it. As you turn off the swap partition, its contents are forced back into memory. “swapoff /dev/sda2” to turn it off, and then “swapon /dev/sda2” to turn it back on.

Mailserver Errors

At work, I have to deal with a ton of mail bounces. We get hundreds a day. Automating the process has been a bit of a side project of mine, but it’s been slow going, for the simple reason that mailservers don’t use standard codes. I’m able to catch something like 50% of mail, but that’s still a lot that gets missed. Some is base64 encoded, and in others, people just thought they’d get cute and use wholly custom text to indicate the problem. When you’re AOL I can write a filter to match “MAILBOX NOT FOUND” even though there’s no 500-level error. When you’re joesdiscountplumbing.com, I’m not going to waste my time. (VERP kind of solves this, except that Exchange doesn’t support plus addressing, so we’d need to set up a new MX to catch mail, which means we’d need to send mail from a new subdomain, and it quickly becomes a really involved project.)

But anyway, today I got this awesome bounce message (I’ve removed the user’s e-mail address):


Message from  yahoo.com.
Unable to deliver message to the following address(es).

<REDACTED@att.net_>:
/libexec/ld-elf32.so.1: Shared object "libc.so.4" not found, required by "qmail-local"

For the uninitiated, that’s a baaaad error on Yahoo’s (or maybe AT&T’s?) end. libc.so.4 is the C library. It’s not quite apples-to-apples, but think, “Cannot find kernel32.dll.”

It’s probably something like a corrupted disk in one of hundreds of servers, but it certainly stood out from all the other “Mailbox unavailable” messages that came in. (I do see a surprising amount of mailservers that give ridiculous errors in some situations, like putting the full SQL query used to try to find the account in the bounce e-mail, or complaints about file permissions.)

Twitter

We talked a bit at work tonight about Twitter. It have a release to push out in an hour, so I’m basically killing time. (Waking up after a 90 minute nap is murder.) So be warned that this post may be a bit rambling. This post is really just a collection of disparate thoughts relating to Twitter.

Immediately after the meeting discussing Twitter, we all went back to our desks and added everyone as friends. It was kind of neat, in that I found one person, looked through their followers, found some more, and pretty quickly had everyone in our office who’s on Twitter. (And not just peers, but our CEO and CTO.) Others presumably had the same idea. My follow counts doubled in about 10 minutes. (I seriously worry that Twitter is going to flag us as spammers.)

A few people mentioned that they still don’t get Twitter. And it occurred to me that it’s kind of like blogging: I’ve been doing it for a long time, and yet I’m still at a loss for what, precisely, the point is. I think of a Twitter as sort of SMS meets blogs: really short status updates. But not, “I’m folding laundry now” status updates: things that your friends might conceivably care about.

I never liked the term “followers,” nor was I too keen on “tweet” or any of its conjugates. Tweeting sounds ridiculous, and makes the action sound even more frivilous than it is. “Followers” are people who blindly immitate you. Kim Jong-Il has followers (albeit mostly coerced). I want to have friends, not followers. I can’t bring myself to say, “I have 24 followers.”

I find Twitter way too complex. For something that’s just 140-character blobs of text, what is the deal with TweetDeck? There are 14 icons across the top, three panes, and it’s practically filling one of my monitors. Plus, whenever anyone posts, it makes the type of sound I would expect to hear if I was being beamed up into a UFO, which startles me every single time it happens. (The TweetDeck noise, that is.)

They have a simple web API, so I’m giving serious thought to writing a little script (Ruby!) that will just pull in new data once a minute and display it in a terminal window. That’s all I want. A simple little interface. TweetDeck is way too intense. I used Syrinx previously on the Mac, but it was really wide. Another program (Twitterific maybe?) gave no option but to always sit on top of your other windows, which doesn’t work well when you have a job and are expected to occasionally do some work there.

People use Twitter in various ways. Most of the people I follow are friends, and they post little notes. It’s neat having coworkers doing it, because it sort of gives me a sense of what people I don’t interact with everyday are up to. There are also a lot of ‘power users’ I follow. The problem is that I’m interested in maybe 10% of what they post. I don’t want to stop following them and miss out on the good 10%, but I don’t want to have to deal with the 90%, either. I do think I should stop following a few, like news companies (cool for a while, but posting way too frequently) and snoopdogg. (Hilarious for a day, then really tiresome.)

The more I think about it, the more I like the idea of a little Ruby script. It’d be a good way to learn the API, and an even better way to master Ruby. Plus, when I eventually had the free time, I could play with XCode and build a slick GUI around it, keeping it a minimalistic tool. It kind of ties into my long-standing desire to just have a little textbox (think IRC) that would just scroll various “events” that I might care about. I think I’d put in e-mails to my personal mailboxes, too.

Burden of Proof

I started to write out a question on my favorite Ask MetaFilter, asking if Obama’s citizenship has ever been concretely proven. There are still some who call it into question.

But as I wrote the question, I started to realize how absurd the argument was. Right about the time he was being labeled a Muslim, people started “pointing out” that he was born in Kenya and thus not a citizen. Nevermind the fact that his father was practically estranged his whole life, or that he was born in a Hawaiian hospital to a mother from Kansas. So Obama’s team obtained a copy of his birth certificate and posted it online, thinking they’d once and for all prove the rumors false.

Then people argued it was just Photoshopped. Clearly, they’d taken a blank and filled in Obama’s information to make it look like he was a US citizen.

Someone else argued the other day that, when he went to visit his grandmother on the verge of death, he wasn’t seen much in public for a few days, which is evidence that he was using millions in campaign cash to bribe Hawaiian officials.

Someone then pointed out that Snopes has debunked the whole thing, concluding that he is a citizen. But apparently, Snopes is run by a “liberal California” couple, so they’re covering for him.

At what point does the burden of proof shift to the conspiracy theorists? I think it’s fair enough to question the citizenship of someone running for President, though I do find it suspect that the first minority president is the only person to ever jump through these hoops. (Just for fun: Obama was born in Hawaii. McCain? South America, though I don’t think many people seriously doubted his citizenship after he was born on a US military base.)

But after the candidate provides a copy of his birth certificate, I think that if you want to argue it’s Photoshopped and that he bribed the Hawaiian government, the burden of proof shifts, and suddenly it’s on the conspiracy theorists to make the case. At the very least, I’m through trying to have a rational conversation about it. (Did you know that Sarah Palin is actually a martian, and a convicted serial killer? It’s true. She used her power as governor to destroy the records of the hundreds of people she murdered, and then used taxpayer dollars to have surgery to appear to be a human. Bet you can’t prove me wrong!)

In Soviet Russia

Failblog posted this amusing homework assignment, and someone in the comments pointed out that if you Google all five of the terms, you get taken to this page [spoiler!], which probably posted it originally and has access to the answer key. (The assignment is apparently from Russia, yet in English.)

I won’t post the real answer (which I argue is wrong), but by far my favorite (inaccurate) one:

They’re all completely screwed.
The penguin is standing in front of a train.
The boot has no foot in it.
The train has no tracks.
The whale is stranded out of the water.
The piano has no bench (nobody wants to play a piano without a bench…)

What do you think the commonality is?

Voting

A few years ago I took an interest in electronic voting machines. As a computer geek, it seemed odd to me that most fellow computer geeks were opposed to the idea. Computers make everything better and easier: why oppose them? I soon found the answer: what they make especially easy is fraud. Rigging a paper election is hard. (Although the frequency with which cases of marked ballots are “lost” makes me wonder…) Rigging a computer election is trivial: you just increment a counter.

We’ve known for a long time that something was up. Every time there’s a national election, there’s a flood of news about insane voting machine stories. Some precincts have one candiate getting more votes than there are voters. Some release periodic updates, and people have watched some candidates lose votes as time goes on. Others had thousands of votes cast before the election starts. And still others have concluded that some candidates got negative votes.

I wrote a paper on this once, and it’s even more insane than it seems. One major voting company employs as a head developer a person who was convicted of wire fraud and computer hacking. Another keeps running into trouble because it’s not a US-owned company. Another sued to prevent the disclosure of security testing results. Oh, and one of the biggest companies had a bunch of e-mails leaked in which developers and executives basically talked about how how they didn’t need security so that people could go in and change things in the backend easily.

The whole think reeked, but it was kind of like some politicians: you were never sure if they were ridiculously inept or if they were truly corrupt. But now it’s getting a little clearer: election fraud is rampant.

Okay, checking out the BradBlog.com site a bit more… Diebold is one of the big voting machine makers, but also a prominent ATM manufacturer. This article is downright scary: Windows malware specificically targeting ATMs. The details are sketchy, but it sounds like it basically reads the data off of swiped cards and sends it to some nefarious entity. Instead of using a hardware “skimmer” on an ATM, they put a virus on the ATM to make the ATM itself the skimmer.

Some things aren’t meant to be computerized. I think voting machines are one of them.