Custom LogFormat with Apache

Posting this in the hopes that it’ll help someone at some point….

Using Apache (Apache2 in my case, but I’m not sure it matters), you can customize the format for log files like access_log. Apache has a good page describing the variables you can use. But it doesn’t tell you everything you need to know!

The first question is where you put it… You can just specify it in httpd.conf (I put it near the end, but I don’t think its placement matters terribly, as long as it’s not in the middle of a section. It doesn’t go in any directives or anything. You can also insert it inside a VirtualHost directive if you only want it to apply to those. (Don’t put it inside a Directory directive!)

The second thing is something that’s not really specified anywhere: specifying a LogFormat without then specifying a CustomLog directive accomplishes nothing! I wanted to keep Apache logging in the default directory (/var/log/apache2/access_log on Gentoo), so I just set the LogFormat to something I wanted. And nothing happened.

You specify the format in CustomLog as well, so it’s handy to use LogFormat to assign a “nickname”:

LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"" n1zyy
CustomLog /var/log/apache2/access_log n1zyy

The first line sets the “n1zyy” ‘nickname’ to refer to to the format I specify. The next line sets a “custom” log file (in this case, it’s the same as the default, but I digress. It won’t work if I don’t specify it.) Then I tell it to use the format named “n1zyy.”

Once this is set up, you want to reload Apache, since it won’t notice your changes until you do.

Ecstatic

In the most recent polls, Obama is leading narrowly in New Hampshire. And it’s practically a banal phrase at this point, but Iowa is a crapshoot: the “big three” (Edwards, Clinton, and Obama) are pretty much tied. Right now it looks like Edwards is leading, which people thought was unlikely. Thus I’m not too worried at the moment about Hillary’s triumphs in other places.

But for the first time in a while, I’m feeling really excited. This could actually happen!

I’m starting to get interested in the Republican primaries as well: they’re seeming pretty fragmented. Romney and Rudy both have big leads over each other in many states, but McCain and Huckabee are notable contenders in some states, too. (Somewhat humorously, at least to me, Romney has a pathetic 7% in Massachusetts, although the poll is ancient. Someone ought to do a new poll of Massachusetts voters.)

Plans are still up in the air but I may well end up volunteering over at the Obama headquarters later today. The nation is watching us, and I don’t want to sit by idly in the process. We can do this!

High Dynamic Range

I’d been seeing a lot about HDR, or High Dynamic Range, photography. In layman’s terms, the dynamic range of a camera is the range from the darkest to the lightest parts a camera can record in one shot. The problem is that the dynamic range of cameras doesn’t match real life that often.

Long ago, photographers found a halfway decent solution: graduated filters. Basically, you stick a filter in front of the lens, with part of it darker than the rest. It’s great if, say, you want to take a great picture at the beach with both foreground detail and the sky properly exposed.

With computers, though, there’s been another photo. You take a series of bracketed shots: one or two for the sky, one or two for the foreground, etc. Some people have been known to stitch together close to a dozen. Having a tripod helps tremendously here, since the images need to be pretty much exactly the same besides exposure.

Strictly, HDR requires more than a monitor can really display, but a technique called tone mapping is often used. The basic premise is to take the “good” parts of each shot in a bracketed series and stitch them together. Photoshop CS2 and newer has an HDR utility, though I’ve been pretty unimpressed with the results. Today I started playing around with an Open Source tool called Qtpfsgui. It’s even cross-platform! It supports multiple algorithms for doing tone mapping, too.

Overall, I’m still not that happy with the results, but it’s a start. Here’s a ‘normal’ shot of the beach, taken on Cape Cod yesterday:

Beach

You’ll note that the foreground (e.g., the bench) is too dark, yet the sky is too light. It’s a good illustration of insufficient dynamic range.

Luckily, I knew in the back of my head that I wanted to try my hand at HDR photography, so I saw it as an opportunity. I set my camera to meter -2 to +2 EV, to try to cover the full range. The end product:

Fattal Algorithm

It displays a very common pet peeve of mine with HDR photos: it looks entirely unrealistic. Absurd, even. I think part of it’s that it’s just overdone, and that the contrast is jacked way up. I want to play around with it more and see if I can get a more natural product. So far, no luck. But, at least in a technical sense, it’s an improvement over the first image.

I’d like to see HDR come a little further, so that HDR photos don’t have the same, “Whoa!” quality that a scary old lady with way too much makeup has. I don’t think the limitations are entirely technical at this point, either.

Geek

We’ve been having a lot of intermittent network problems at home. Periodically, our Internet cuts out. At first I assumed it was our ISP–it’s no longer Adelphia (run by pharmacists), though–but subsequent research indicated that it wasn’t our ISP’s fault: our router was going down.

My dad set it all up, so I wasn’t too sure how things went. I was pretty confident that we were just using a generic store-bought broadband router, though, so I found it strange that it would be drifting in and out. It turns out that I overlooked something about the router: it’s being held together with duct tape.

I’d already been intrigued by OpenBSD’s pf, so this seemed like a sign! I commissioned an old desktop system, loaded OpenBSD up on it, and went to work configuring it. OpenBSD was just more different from Linux than I expected. It asks you if you want to let OpenBSD use the whole hard drive. I said yes, and thought, “Wow, this is just as easy as Ubuntu!” But it turns out that this was just the first stage. After this, you have to set “disk labels,” which are sort of like partitions but ambiguously different. The syntax is obscure, the purpose is obscure, and so forth. Then I had to configure the network. NICs are named by the drivers they use, so instead of eth0 and eth1 (for Ethernet), I have rl0 (Realtek) and dc0 (who knows).

I was also extremely confused trying to set up routing. Long-term, it was going to be the router, but short-term, it needs to know about our existing router so that it can connect and download the requisite packages.

So I finally got it all set up. I also installed MySQL (unnecessarily, it turns out), Apache, and PFW, a web-based configuration tool for pf. I ended up not using PFW, because my understanding of pf is so bad that I’m basically relegated to copying-and-pasting rules from websites into the configuration file.

Even using pf is confusing. It’s called pf, but typing “pf” at the command line doesn’t do anything. It turns out that you control it with a tool called “pfctl.” You can do pfctl -e to enable pf, and pfctl -d to disable it.

As I tried to tweak the firewall/routing rules, I’d periodically “restart” pf by disabling and then re-enabling it. I wasn’t sure if it read the rules “live” or if a restart was needed. It turns out… neither! The rules are stored in memory, but restarting pf doesn’t flush the rules. You need to pass pf some more arguments to tell it to flush the cache and read them anew from its configuration file.

After a few more hours of work, I thought it was all set up. Both NICs were configured, the external one to get an IP over DHCP, and the internal one with a low fixed IP. I had a complex set of rules, doing NAT, filtering traffic, and using HFSC for prioritized queueing. (HFSC seems completely undocumented, by the way. I took my tips from random websites.) It seemed very impressive: I prioritized ACKs so that downloads wouldn’t suffer if our outbound link was saturated. (Aside: it really doesn’t make sense to do queueing on incoming traffic, since the bottleneck is our Internet link, not our 100 Mbps LAN.)  I also afforded DNS, ssh, and video game traffic high priorities, but allocated them a lower percentage of traffic. I even figured out the default BitTorrent ports and gave them exceptionally low priority: if our line is fully saturated, the last thing I care about is sharing unnecessary data with other people.

And there are other neat features. It “scrubs” incoming connections, reassembling fragmented packets and just eliminating crap that doesn’t make sense. It catches egregious “spoofing” attempts and discards them.

I hooked up the second LAN connection to test it out, rebooted, and… waited.

It never came up. Well, it did come up. The computer’s running fine. Both network cards show up with the switch. Doing an nmap probe of our LAN, I see one strange entry. It’s actually pretty mysterious: it has no open ports, and attempting to ssh into it just sits there: it doesn’t send a connection refused, but completely ignores the incoming packets, leaving my poor ssh client sitting there waiting for a reply, having no clue what’s going on.

In a nutshell, it seems that I just built a firewall/router that’s so secure that I can only find one of its two cards on the network, and I can’t even try to log into it. Let’s see you hack that! Of course, this does have some issues. For example, I can’t use it.

I haven’t lost hope yet: I have a keyboard and monitor so I can log in on the console and try to do some tweaking there. (You can’t firewall off the keyboard.) It’s just not very encouraging to think, “Alright, let’s reboot and make sure it works as flawlessly as I think it will” and then have the darned thing not even show up on the network.

Benazir Bhutto

I confess to being ignorant enough to have not even heard of her, but Benazir Bhutto was a really interesting figure.

Now here’s an interesting video. You learn a few things. The first is that she speaks fluent English. The second is that she was widely aware of plots to kill her, and fingers a number of suspects in the video.

But the person who posted the video makes another interesting point. At one point she speaks of Osama’s son. Later on, she fingers a man “who killed Osama bin Laden,” an assertion which doesn’t seem to phase the interviewer.

The rumor’s existed for a while, but has generally just been peoples’ gut feelings and such. Now I’m intrigued.

…and a Happy New Year!

(Okay, it works in chronological order, but I display newest on top… So just pretend my title complements Kyle’s.)

I wasn’t planning on blogging about my Christmas presents, but Kyle did and I decided to save some stuff for a new post.

I was much relieved when the former bishop of Turkey brought a Rebel XTi to replace my 10D (RIP, buddy; I loved you for the short time I knew thee). Although it’s technically a lower “class” of camera, the XTi is really an upgrade to the 10D in all ways except size and weight, so I’m quite pleased. (I “lost” ISO3200, but it was so noisy that i don’t miss it.) Not only is it a higher resolution (and a bigger LCD!), but Canon introduced an awesome new feature: an ultrasonic “duster” for the sensor that runs every time you turn the camera on or off. It’s too soon to tell, but it’s seemingly pretty effective at making sensor dust a problem of the past.

Along with it was a 50mm f/1.8 lens… I was a bit concerned at first, because it’s an effective 80mm with the FoV crop, but it’s turned out to still be an ideal length. The f/1.8 aperture affords me two great abilities: one is to take pictures in comparatively dark places without relying on flash, and the other is the ability to throw the background way out of focus, achieving “bokeh,” a fabulous effect.

Holly

I should note that, in the past few days, I re-shuffled things on my computer, re-installing Ubuntu on a clean partition and getting Compiz working. I’m hoping to use Xen to run my Windows installation, but I haven’t gotten Xen and my desktop environment to play nicely yet. I backed up my 500 GB “backup” drive, reformatted and repartitioned it (in a sane manner this time), and then moved everything back onto it in a more organized manner. I also set up an old stereo I had almost forgotten I owned. So it was practically Christmas even before today.

We also got a Wii for the family, along with Guitar Hero 3. Trying to get my parents using it, I realized just how steep the learning curve is: they’ve probably sunk a couple hours into practice and are just now finishing songs. It was the same way for me, too, just a long time ago. In a way, I kind of wonder why people bother: if you spend half an hour and get nothing but the crowd booing you, it’s really not encouraging to keep going. GH3 on the Wii is interesting–you snap a Wiimote into the back and use that, making it a wireless guitar. (Woot!) As an added bonus, the sounds when you mess up come out the controller and not the TV, which would be very helpful in multiplayer mode.

Guitar Hero 3 for the Wii

The Wiimotes now ship with this silly-looking “skin” for the controllers. I’m not sure whether it’s to protect the controllers (which practically explode if they get flung into a cinder block wall) or to protect people (who, presumably, do not like being hit in the head with game controllers), but it’s probably a good idea either way… They just look a bit goofy, is all.

New Wiimotes

I also got some great books… I’ve started several, and am having a hard time deciding whether I should keep up status quo (reading a chapter or two from one and then coming back and picking up another book and continuing that), or read them sequentially. Current must-reads on my nightstand* include my (signed!) copy of The First Campaign by Garrett Graff, an expert on blogging and politics; Tim Ferriss’ The Four-Hour Workweek (pre-review: the little bit I’ve read is fascinating, but between the book and his website, I can’t help but pick up on a bit of ego?); “Why Are All the Black Kids Sitting Together in the Cafeteria,” an interesting (or so it looks; I haven’t gotten far in yet) look at race relations in America by Spelman College President Beverly Daniel Tatum; and Naked Economics, a thin paperback “Undressing the Dismal Science” by Charles Wheelan: it looks like the type of book I wish I’d had when I was taking Economics.

I also received a nice vacuum. Ordinarily, I wouldn’t be too excited about a vacuum cleaner. Think of the, “Oh, it’s more clothes?!” you felt as a child. I think that’s how most people would feel upon receiving a vacuum cleaner. Especially college-aged guys. But you should see the floors in my dorm room… The vacuum was among my favorite gifts this year. Our floor at school gets vacuumed about once a month. It needs to be vacuumed about thrice a week. So it’s going to be a huge improvement.

* Full disclosure: I don’t actually have a nightstand, but I didn’t think it was too egregious of a lie to not say that the books are actually split between my desk and the side of my bed. But in case anyone wants to try to accuse me, there it is: I don’t have a nightstand.

Advice

I learned two valuable lessons today:

  • Don’t ever create a 500GB FAT partition. No matter how good of an idea it seems, don’t do it. (Not terribly different is the advice, “Don’t ever create one big 500GB partition.”)
  • Mounting a filesystem as “msdos” is not the same as mounting it as “vfat” in Linux. msdos is still constrained by the 8.3 naming system. vfat is not. Unless the disk was literally written with MS DOS, don’t use msdos. It’ll work okay, but boy are you screwing yourself if you make backups with it mounted as msdos. (Fortunately, I realized this before wiping the drive.)

An Uncontrollable Urge

A few years ago Andy and I ran a hosting company. It never got that far, but it was fun, and also a learning experience.  Today I’m finding that I can’t get the idea of starting it again out of my head. The problem is that, this time, I’d want to start it big.

There are a bunch of technologies that I find downright exciting:

  • Old racks full of blade servers are hitting the used market. And by “old” I mean dual 2-3 GHz Xeons, a gig or two of RAM, and hard drives that still rival what hosts are renting in dedicated servers. I’d probably want to put in new drives, but the machines are cheap and they’re plentiful.
  • Boston has a number of good data centers, and all the big Tier 1 providers are here. That there seem to be no well-known hosting companies out here is frankly kind of surprising. You have no idea how badly I want to pick up a couple racks in a colocation facility, and pull in a couple 100 Mbps lines.
  • cPanel looks like it’s matured a lot since I last used it, and it has some good third-party stuff such as script installers. It looks like it remains the number one choice in virtual hosting.
  • Xen is downright exciting. It permits splitting a physical host into multiple virtual machines. With the advent of chips with hardware virtualization support from both AMD and Intel, it now runs with very little overhead. It used to require extensive modifications to the “guest” OS, so that only modified versions of Linux worked. With newer processors, though, you’re able to run machines without them having to know they’re in a virtual machine, opening up options. You can run Windows now. The virtual dedicated server / virtual private server market is growing. (Xen also supports moving hosts between physical servers, which has a lot of nice applications, too!)
  • OpenBSD’s firewall, pf, continues to intrigue me for its power. I just found PFW, a really spiffy web GUI for managing pf. Not only does it do basic firewall stuff, but it’s got support for prioritization of traffic / QoS, and for load balancing. I’m probably just scratching the surface.
  • I’ve spent years honing my admin skills and improving server performance. Improved performance on a shared server, of course, means more clients per server, or more money.

I’m wholly convinced I should start a Boston hosting company. I just need $100,000 capital or so. (Santa, do you read my blog? Do you fund businesses? I’ll give you partial equity.)

Knots in My Stomach

Thanks Rusty for finding the Electoral-Vote.com website, something I’d forgotten about from the 2004 election. The data is in a bit of a confusing layout… Disregard the 2004 map and the first little table. He then has a comprehensive list of polls state-by-state.

My eyes are on Clinton:Obama. And I seriously have knots in my stomach here. Clinton is winning by at least 10% in most places. Arizona is 44% to 14%. In his home state of Illinois, Obama’s winning 37% to 33%.

The good news! Iowa, a key state, is slightly favoring Obama. But really, it’s a crapshoot: Obama, Edwards, and Clinton are neck-and-neck. Romney and Huckabee lead the Republican primary. At this point in time, though, my main concern is on the Democratic primary.

Here in New Hampshire, Obama’s trailing, 26% to 38%. This is not good. We’re #2 after Ohio.

Oklahoma’s weird. Obama’s got 13%, with Clinton and Edwards tied at 29%. (Don’t get me wrong: Edwards is good, but I don’t think he has a chance right now.)

The Republican one is interesting to take a gander at, too. In some places, Huckabee’s an also-ran. In Arizona, he got 3% of the votes. Once. In Iowa, he inches past Romney to take first place at 28%. Surprisingly (to me, at least), he’s doing the exact same thing in New Hampshire. With a quick skim (admittedly, much less than I’ve afforded the Democratic primary), it looks like Giuliani is king of the Republican race.

But a few thoughts:

  • I think the odds of Edwards winning the primary are slim. But he carries a substantial margin in some places. If he were to drop out and endorse Obama, the impact would be considerable. I worry that most of his fans would support Hillary, though.
  • I think we need to review the statistics after the Iowa caucus (January 3) and the New Hampshire primary (January 8). Everyone’s watching these, and the results will have a big impact. A strong lead by Obama may pull out some undecideds. Or, a strong lead by Clinton may freak out some people who will vote for Obama just to vote against her. (While I’d back her if she were our nominee, she is not my preferred Democrat, if you can tell.)
  • My super-early-money is on Clinton vs. Giuliani. And this concerns me greatly, because people voting on first impressions will probably favor Rudy without really doing a lot of research. (It also concerns me because I don’t particularly like either of them.)
  • The Republicans are getting weird results: Giuliani wins some places, Romney wins some places, McCain’s got a few wins (probably the least), and Huckabee, who I initially thought was the Kucinich of the Republicans, is actually leading in quite a few places. I’m really not sure who’s going to get their nomination.
  • As we saw in 2004, polls can be flaky. (I twice typed “pols” instead of “polls.” Freudian slip?) So this doesn’t necessarily mean anything.

One-sentence conclusion: It’s too soon to really have any idea how things will go, but Clinton has a discomforting majority in many states.

A few parting thoughts:

  • Read up on the Iowa caucus process if you’re not familiar. It’s quite foreign, really.
    • Apparently, only once in history (or once in five, put differently: an important distinction!) has the Straw Poll winner not matched the Iowa caucus winner. And this year’s Straw Poll winner was Romney. Both Giuliani and McCain screwed everything up by blowing the event off, and thus polled very poorly. I don’t know what this means: this might still tick off Iowa voters, tanking Giuliani in Caucus as well. But it also means that the data is probably skewed away from them right now, and if Iowa voters don’t have a vengeance, they may take votes away from Romney.
  • The Iowa Caucus is less than two weeks away, and the NH primary is less than three. Pay more attention to the statistics then.
  • Vote!