Time

So I’ve mentioned before that I run an NTP server. Stratum 2, which means it gets its time from a “Stratum 1,” which is set directly to something reliable. The main goal of NTP is to keep clocks in sync, and it’s pretty accurate, down to a fraction of a second, which is more accuracy than most people need. All of my computers will now agree on the time down to a second.

The ultimate source, of course, is the atomic clock. But there isn’t an atomic clock, per se. There’s actually an array of them, each using cesium or hydrogen as an atomic reference. Collectively they form “the” atomic clock, which is used as a frequency standard.

It’s all well and good to keep your computer clock (and wristwatch, and microwave, and oven, and wall clock…) synced within a second, but some things need more accuracy. The USNO (US Naval Observatory, in charge of maintaining the atomic clock system) explains one common scenario well: systems for determining one’s location, such as GPS and LORAN “are based on the travel time of the electromagnetic signals: an accuracy of 10 nanoseconds (10 one-billionths of a second) corresponds to a position accuracy of 10 feet.” There are also lots of other scientific uses for extremely precise time, many of which I couldn’t even begin to understand the basic premise of. But suffice it to say that there are actually a lot of times when knowing the time down to the nanosecond is important.

Things like NTP don’t cut it here. You can get down to the millisecond, but you need to be about a million times more accurate. (A millisecond is a thousand microseconds, which is a thousand nanoseconds.) So how do you keep the exact time? It turns out that there are actually several ways. One way (decreasingly common) was to keep an atomic clock of your own. You can buy a “small” (the size of a computer…ish) device that has cesium or hydrogen or rubidium inside of it, which keeps pretty accurate time. Over time it’ll wander, but at least short-term, it’s quite accurate.

One of the first ways is WWV, a shortwave radio station. (And it’s Hawaiian sister station, WWVH.) They run continuously, disseminating the exact time via radio as observed from the atomic clock system. In the past I’ve synced my watch to this source. More notable, in a behind-the-scenes type of way, is WWVB, a low-frequency (60 kHz) radio broadcast. This is what all your “atomic wall clocks” sync to. (Incidentally, I’ve read that most of them are fairly cheaply built, meaning that their time is really not accurate to more than a second.) Another interesting sidenote is the deal with their antennas: a quarter-wavelength antenna at such a low frequency is 1,250 meters tall, or about 4,100 feet (nearly a mile). But with some wacky designs they can overcome this (although pouring 50,000 Watts into it also helps).

The problem with “straight” receivers for WWVB, though, is that you have to figure in the time it takes for the signal to reach you, which is rarely done all that well (if at all). Instead, a more common technology is used: GPS.

It turns out that GPS carries insanely accurate time. Wikipedia has a really good article on it. Each GPS satellite carries an atomic clock onboard, and people on the ground keep it synced (with nanosecond accuracy) to the atomic system. There’s some funky correction going on to keep things perfectly accurate. GPS has a claimed accuracy of 100 nanoseconds, although people have found that it’s actually about ten times better, down to 10 nanoseconds or so.

As an aside, GPS in general is an interesting read. There’s a lot more going on than meets the eye. I recently dug up an old GPS and wondered if it needed an “update” to get new satellite positions: with ham satellites, we get periodic updates for our tracking software to account for changes in their path. GPS has a neat solution, though: the satellites broadcast this data. Actually, more accurately, they broadcast all the data for all the satellites, so that seeing one satellite will fill you in on the whole setup. There used to be Selective Availability, basically a deliberate introduction of error into the signal. The premise was that we didn’t want enemy forces using it: imagine a GPS-guided rocket, for example. So we introduced error of about 30 meters for a while. Ironically, it was ended because our own troops (before Iraq) couldn’t get the military units, so they were just buying off-the-shelf civilian units and incurring the decreased accuracy. So Selective Availability has been turned off, and there are indications that it was permanent. A third interesting tidbit is that the GPS satellites carry much more than might meet the eye, including equipment monitoring for nuclear detonations.

The timekeeping problem is what to do when you get the time at the GPS, though. High-end GPS units will provide a pulse-per-second signal, which you cna hook up to a computer via serial, and achieve great accuracy. But there are all sorts of considerations I never thought of. Between the time it actually charges the pin and the time the operating system has processed it takes a little bit of time, os there are special kernel modifications available for Linux and BSD to basically get the kernel directly monitoring the serial port, to greatly speed up its processing. I also discovered the Precision Time Protocol (commonly known by its technical name, IEEE 1588), which is designed to keep extremely accurate time over Ethernet, but apparently requires special NICs to truly work well.

I’ve also learned another interesting tidbit of information. CDMA (which is a general standard, not just the cell phone technology that Verizon uses) apparently requires time down to the microsecond to keep everything in sync, such as your multiple towers and all the units (e.g., phones) in sync and transmitting at the right times. So the easiest way to keep all of their towers in sync to a common standard was to put a GPS receiver at each tower and sync the system to that. Thus CDMA carries extremely accurate time derived from GPS, which has led to some interesting uses. It’s hard to get a GPS signal indoors, so they now make CDMA time units–they sit on a CDMA network in receive-only mode, getting the time but never taking the “next step” of actually affiliating with the network. This lets people get GPS-level accuracy inside buildings.

Digital Photo Recovery

I just discovered PhotoRec, a tool for recovering digital camera images.

For the non-geeks, a quick basic background…. When you save a file, it writes it to various blocks on the disk. Then it makes an entry in the File Allocation Table, pointing to where on the disk the file is. When you delete a file, the entry is removed from the File Allocation Table. That’s really all that happens. The data is still there, but there’s nothing pointing to where on the disk it is. This has two implications. The first is that, with appropriate tools and a little luck, you can still retrieve a file that you’ve deleted. (Whether this is comforting or distressing depends on your perspective…) The second is that, with no entry in the File Allocation Table, it’s seen as “free space,” so new files saved to the disk may well end up getting that block. It’s technically possible to recover stuff even after it’s been overwritten, but at that point it’s much more complex and much more luck is involved.

Last night we went out to dinner… We took lots of photos, but some were deleted. So I figured PhotoRec might recover them. So I gave it a try.

The filesystem shows 163 photos. After running PhotoRec, I have 246 photos. What’s odd is what photos I have. It’s not the ones from last night. They’re scattered from various events, and several are from almost two months ago.

This does leave us with an important tip, though: if you delete an essential photo, stop. Each subsequent thing you do to the disk increases the odds of something overwriting it. In a camera, just turn it off. Taking more photos seriously jeopardizes your ability to recover anything.

In my case, I didn’t have anything really important… I just wondered how it would work. And I got strange results for recovered files. (Which has me wondering a lot about how its files get written out to disk, actually.) But it’s good knowledge for the future. (By the way, PhotoRec runs under not just Linux, but also, apparently, Windows, and most any other OS you can imagine.)

Closed Source

As much as I love open source software, I tend to shy away from the die-hard “OpenSource or bust” people. I use closed-source (“restricted”) drivers when need be, and they usually work better, since the vendors can optimize them.

I’m quite frustrated, though, with ATI… The closed-source fglrx drivers give good performance, but have some major problems. Namely, they just don’t work with Xen. I’ve been looking to set up some virtual machines, but I have the choice of using VMs or having video drivers…

And hibernate / software suspend has never worked. It turns out that this is also a known bug caused by using the closed-source fglrx drivers.

It turns out that the Ubuntu kernel team is aware of both of these, and trying to find ways to fix it. But the problem lies in a closed-source module, so their hands are tied.

Argh!

Do you have the time?

I’ve been running an NTP server on this host for quite some time now. But as of yesterday, I’m a member of the pool.ntp.org group. pool.ntp.org is a round-robin-ish DNS service where requests for pool.ntp.org are given IPs from a huge block of listed nameservers, balancing the load across a pool of about 1,500 NTP servers across the world. The official “entry” for this server is my IP (72.36.178.234), but ntpd is actually listening on all IPs right now, so using blogs.n1zyy.com or ttwagner.com will work.

I’m currently synced to Stratum 2 servers, but I think that, after I finish up some open tasks (“real work,” versus playing with time servers), I’m going to look at requesting permission to sync to Stratum 1 servers. (Stratums, err, strata, are basically tiers. “Stratum 1” refers to a server directly connected to something like a GPS (which obtains extremely accurate time: having the correct time is an important part of how GPS works, so GPS actually broadcasts the time from the atomic clock) or from WWV (transmitted over HF radio). Stratum 2 servers get their time from Stratum 1 servers, and so on. As I sync to a network of stratum 2 clocks, I become a stratum 3 server. Moving up a stratum generally implies more accurate time, as there are fewer intermediaries to skew results. (Although we’re talking milliseconds of difference.) There aren’t an awful lot of stratum 2 servers, so syncing to a stratum 1 server would help to round out the stratum 2 list. (It would be fun to become a stratum 1 server, but as a stratum 2 host says of his data center, “they’re not going to let me drill a hole in the ceiling to run an antenna [for the GPS] to the roof.”)

For those of you with UNIX systems, take advantage of this! You can sync to me directly (72.36.178.234), or indirectly (the pool.ntp.org cluster). (Windows can sync to an NTP server as well, it’s just not a standard feature.)

Web Design

I’ve redone ttwagner.com. It’s no longer a random integer between 0 and 255, but instead, a decent-looking site. I’ve integrated some of the cool things I’m hosting there as well. I came across a few interesting things I wanted to point out.

The world DNS page is incredibly intensive, and, since it’s not dynamic, there’s no sense in “generating” it each time. So I used the command wget http://localhost/blah/index.php -O index.html to “download” the output, and save it as index.html in the web directory. Viola, it serves the HTML file rather than executing the script.

But the HTML output was frankly hideous. The page was written as a, “You know, I bet I could do…” type thing, written to fill some spare time (once upon a time, I had lots of it). So I’d given no attention to outputting ‘readable’ HTML. It was valid code and all, it just didn’t have linebreaks or anything of the sort, made it a nightmare to read. But I really didn’t want to rewrite my script to clean up its output so that I could download it again….

So I installed tidy (which sometimes goes by “htmltidy,” including the name of the Gentoo package). A -m flag tells it to “modify” the file in place (as opposed to writing it to standard output). The code looks much cleaner; it’s not indented, but I can live with that!

I also found that mod_rewrite is useful in ways I hadn’t envisioned using it before. I developed everything in a subdirectory (/newmain), and then just used an htaccess override to make it “look” like the main page (at ttwagner.com/ ). This simplifies things greatly, as it would complicate my existing directory structure. (It’s imperfect: you “end up” in /newmain anyway, but my goal isn’t to “hide” that directory, just to make the main page not blank.)

I’ve also found I Like Jack Daniel’s. (Potential future employers: note the missing “that” in that sentence, which changes the meaning completely!) The site is a brilliant compendium of useful information, focusing on, well, Apache, PHP, MySQL, and gzip, generally. The “world DNS” page was quite large, so I decided to start using gzip compression. He lists a quick, simple, and surefire way to get it working. (The one downside, and it’s really a fundamental ‘flaw’ with compression in general, is that you can’t draw the page until the whole transfer is complete. This has an interesting effect as you wait for the page to load: it just sits there not doing much of anything, and then, in an instant, displays the whole page.) It may be possible to flush the ‘cache’ more often, resulting in “progressive” page loading, but this would be complicated, introduce overhead, and, if done enough to be noticeable, also defeat the point of compression. (Extreme example: Imagine taking a text file, splitting it into lots and lots of one-byte files, and then compressing each of them individually. Net compression: 0. Net overhead: massive!)

Gnome’s Trash Folder

Gnome (the desktop environment, not the creepy garden fixtures) has a “Trash Folder,” which is the functional equivalent of the Windows Recycle Bin. This is slightly problematic for those of us who are used to file “deletion” on Linux instantly unlinking the file, who don’t expect that the files are being moved somewhere. Especially as the developers made the icon a little more obscure, putting it in the very bottom right corner of the taskbar, where I overlooked it for quite some time. Now that I know it’s there, it’s easy to purge, just like I empty the trash bin.

But for geeks like me, you’ll be happy to know that the “Trash Bin” is nothing more than a directory, ~/.Trash. Thus you can empty it fairly easily, with “rm -rf ~/.Trash/*”  And you can use all the routine Linux filesystem tools: “ls” to show what’s in it, and “du -sh ~/.Trash” to list its contents and show the total size.

Update: Newer distros are putting it in ~/.local/share/Trash/files/ instead of ~/.Trash

Flash Plugin for Ubuntu

In the hopes of helping someone else…  Ubuntu has two options for a plugin for Firefox to play Flash. There’s the open-source Gnash and “the real thing” from Adobe. Of course what Adobe provides is closed-source, so there seems to be a tendency to push Gnash. But having used Gnash, it’s awful. I give them enormous credit for their work, but when I have a load average of 2.5 playing a 2-minute YouTube video, get lots and lots of dropped frames, and have random 0’s (a big ‘zero’) superimposed over the video, it’s alpha-level code. It’s not usable.

In theory, Ubuntu makes it easy to install the proprietary one, too. They can’t distribute the player itself, per Adobe’s license terms. So they have a package, flashplugin-nonfree, which downloads it from Adobe. The problem is that Adobe updated their version, so the script fails, citing an MD5 mismatch. (The file’s signature has changed, since it’s, you know, a different version.) Worst of all, it doesn’t handle this error appropriately: it tells you that the package installed successfully, when, in actuality, it aborts due to the error.

This happened in early December. The Ubuntu developers have been squabbling, yet to release an update: apparently the new version doesn’t work in Konquerer. As one person on the developers’ list points out, status quo is the worst case: Flash doesn’t work for anyone. But the developers seem unwilling to use a simple patch to handle the new version, because then Konqueror wouldn’t work. So some grandiose task is under way to try to integrate it with Konqueror, apparently. Meanwhile, no one can use Ubuntu’s package manager to install Flash player.

Fortunately, it’s easy to do yourself. Download the .tar.gz file here, and extract it somewhere. Pull up a terminal, go into the extracted directory, and type “sudo flashplayer-installer,” and follow the on-screen instructions. It’ll take about fifteen seconds. When asked for the location, you probably want /usr/lib/firefox (assuming you use Firefox, and are on Gutsy). You have to close your browser to do this.

Easy Backups on Linux

It’s good to keep backups, especially of servers in remote data centers using old hard drives.

rsync -vaEz --progress user@remote:/path /local/path

In my case, I’m doing it as root and just copying /, although, in hindsight, I think I should have used the –exclude=… option… It doesn’t make any sense to me to “back up” /proc or /dev, /tmp is iffy, and /mnt is usually not desired.

A few notes: I use –progress because otherwise it wants to just sit there, which is irritating.

-a is archive, which actually maps to a slew of options. -z enables compression. Note that this may or may not be desirable: on a fast link with a slower machine, this may do more harm than good. There’s also a –bwlimit argument that takes KB/sec as an argument. (–bwlimit=100 would be 100KB/sec, or 800kbps.)

Using rsync for backups is nothing new, but it’s still not used as widely as it could be. A seemingly-common option is to create a huge backup with tar, compress it, and then download the massive file. rsync saves you the overhead of making ludicrously-large backup files, and also lets you just download what’s changed, as opposed to downloading a complete image every time. It’s taking forever the first time, since I’m downloading about 40GB of content. But next time, it’ll be substantially quicker.

With backups this easy, everyone should be making backups frequently!

Stomatron

I’ve been working on my resume as I seek to apply for a job that’s a neat blend of multiple interests–managing web projects (even in my preferred LAMP environment), politics, and even a management potential. And as I do it, I’m remembering all the stuff I did at FIRST, and reflecting on how much better it could be.

I was “fluent” in SQL at the time, but didn’t know some of the neater functions of MySQL. For example, when I wrote the web management interface to the Stomatron, I didn’t know that I could make MySQL calculate times. So I’d retrieve a sign-in and sign-out time and use some PHP code to calculate elapsed time. This wasn’t terrible, really, but it just meant that I did more work than was necessary.

More significantly, I didn’t know about the MySQL query cache. (Actually, I don’t know when it was introduced… This was five years ago.) Some of the queries were quite intense, and yet didn’t change all that often. This is exactly where the query cache is indicated.

Worse yet, I really didn’t do much with the idea of caching at all. Being the stats-freak that I am, I had a little info box showing some really neat stats, like the total number of “man hours” worked. As you can imagine, this is a computation that gets pretty intense pretty quickly, especially with 30+ people logging in and out every day, sometimes repeatedly. Query caching would have helped significantly, but some of this stuff could have been sped up in other ways, too, like keeping a persistent cache of this data. (Memcache is now my cache of choice, but APC, or even just an HTML file, would have worked well, too.)

And, 20/20 hindsight, I don’t recall ever backing up the Stomatron box. (I may well be wrong.) Especially since it and our backup server both ran Linux, it’d have been trivial to write a script to run at something like 3 a.m. (when none of us would be around to feel the potential slowdown) to have it do a database dump to our backup server. (MySQL replication would have been cool, but probably needless.) If I were doing it today, I’d also amend that script to employ our beloved dot-matrix logger, to print out some stats, such as cumulative hours per person, and maybe who worked that day. (Which would make recovery much easier in the event of a catastrophic data loss: we’d just take the previous night’s totals, and then replay (or, in this case, re-enter) the day’s login information.)

I’m not sure it was even mainstream back then, but our website could have used a lot of optimization, too. We were admittedly running up against a really slow architecture: I think it was a 300 MHz machine with 128MB RAM. With PostNuke, phpBB, and Gallery powering the site, every single pageload was being generated on the fly, and used a lot of database queries. APC or the like probably would have helped pretty well, but I have to wonder how things would have changed if we used MySQL query caching. Some queries (like WordPress’s insistence on using exact timestamps in every one) don’t benefit. I wonder if phpBB is like that. I have a feeling that at least the main page and such would have seen a speedup. We didn’t have a lot of memory to play with, but even 1MB of cache probably would have made a difference. As aged as the machine was, I think we could have squeezed more performance out of it.

I’m still proud of our scoring interface for our Lego League competition, though. I think Mr. I mentioned in passing a day or two before the competition that he wanted to throw something together in VB to show the score, but hadn’t had the time, or something of that sort. So Andy and I whipped up a PHP+MySQL solution after school that day, storing the score in MySQL and using PHP to retrieve results and calculate score, and then set up a laptop with IE to display the score on the projector. And since we hosted it on the main webserver, we could view it internally, but also permitted remote users to watch results. It was coded on such a short timeline that we ended up having to train the judges to use phpMyAdmin to put the scores in. And the “design requirements” we were given didn’t correctly state how the score was calculated, so we recoded the score section mid-competition.

I hope they ask me if I have experience working under deadlines.

Beat the Rush

In case anyone here is interested, I’m hosting a VMware Player image for BlueQuartz, the ‘modern’ GPL version of the old Cobalt RaQ software. A lot of people seem to want a VMware image. I was one of them, until I ended up just creating one on my own.

So grab it while it’s hot! (Read: grab it before I take the time to better throttle download speed.)