As I’ve mentioned before, this “new server” is actually a virtual machine. In the past, Andrew and I each maintained a dedicated server of our own (after I found a great deal years ago and convinced him to sign up for the same package at the same place), but a whole bunch of little things happened in sequence to convince Andrew and I early this year that it made sense all around to move to a single ‘workhorse’ server and run stuff in virtual machines… So instead of each having an aging Sempron with 512 MB RAM, we now share a dual-core 2GHz P4 system with 2GB RAM, and each run our own server inside a Xen virtual machine. (And, incidentally, shortly after we made up our minds to do this, our old hosting company decided to almost double our current prices… The old machines were mediocre hardware when they were new, but a 2-year-old mediocre server is a piece of junk today… So when they increased prices to something that would have been a bad deal when they were new, it made the decision to jump ship even easier.)

Anyway! The old system had 512MB RAM, plus a one-gig swap partition. (Think “paging file” if you’re a Windows user. It’s the same thing.) While this would be downright unacceptable on a Windows Vista desktop with iTunes streaming music and Firefox open with 11 tabs as I have now, plus Photoshop in the background… It’s not bad for a server. The majority of the RAM tended to be semi-free, just used for caching files until something needed it. Apache+PHP+MySQL+postfix+sshd+ntpd certainly used plenty of RAM, but the truth is that 512MB was plenty. And in the extremely rare case that I was low on memory, I had the 1GB swap partition to fall back on: it hurt performance, but it almost never happened.

Since we often have a handful of virtual machines running on this new one, we tend to use small memory sizes: there’s no since in an experimental system with nothing running on it consuming much in the way of system resources. The problem is that this virtual machine got launched running 256MB, which is at the lower-end of what’s acceptable. I’m sitting with about 10MB free, and launching some things (like the wp-admin/ backend!) can push it to the maximum. I also don’t have a swap partition (though it looks possible).

So I ran into something I’d never seen before: oom-killer, the Out-of-Memory Killer. It’s a sort of last resort that the Linux kernel will call, to free up some RAM when there’s absolutely nothing else it can do. (You need at least a nominal amount of RAM free for core functions.) It appears to be carefully structured to be fairly good at choosing the “least bad” thing to kill. (That might be why I haven’t noticed it!) It’s kind of neat to look at a bit, as it’s one of those “last resort” things that I never knew even existed.

Of course, the cool thing with Xen is that you can alter a machine’s memory allocation as it’s running, since the virtual machine’s OS, in the eyes of Xen on the physical hardware, is just another process. I need to look futher into this, as my attempt at raising it from 256 to 384 MB (and setting “max-mem” to 512MB to see how giving a range works) appears to have left my virtual machine with 270MB of RAM, which is an odd result. I think the guest needs a bit of support to know that it’s running inside a virtual machine that has just given it more RAM, though why the memory grew a little bit instead of not at all isn’t entirely clear to me right now…