I’ve mentioned before about how the sync_binlog setting in MySQL can be especially slow on ext3. Of course, I wasn’t the first to discover this; the MySQL Performance Blog mentioned it months ago.

I was reading through some of the slides I mentioned in my last post, and remembered that I’d left sync_binlog off on an in-house replicated slave. You’re able to set it on the fly, so a quick set global sync_binlog=1 was all it took to ensure we flushed everything to disk.

A while later I blogged about dstat and thought to run it on the in-house MySQL slave. I was confused to notice that the CPU was about 50% idle, 50% “wai” (I/O wait). For a box that’s just replaying INSERTs from production, that’s really bad. Below is a graph of load on the system. Care to guess when I enabled sync_binlog?

Performace with and without sync_binlog enabled

Disk I/O does roughly the same thing, but it’s less pronounced, “only” doubling in volume. But the difference is still pretty insane!

We were talking quite some time ago at college about how gay marriage had been legal in Massachusetts for a few years. Despite all the hubbub about it being the end of the world–or the greatest thing since sliced bread–we came to the conclusion that it was a complete non-issue. None of us knew anyone who knew anyone who had ever been to a gay wedding, and even my conservative peers came to not care in the slightest that same-sex couples could get married. It’s gone on for years, and the impact to conservatives who felt threatened turned out to be nil.

And then I remembered that Massachusetts had decriminalized the possession of small amounts of marijuana. Like gay marriage, it drew a lot of criticism, and had many panicked that the sky was falling. About four months (I think?) later, absolutely nothing has happened as a result. It’s had zero impact on my life or those I know. Massachusetts doesn’t suddenly have a drug catastrophe.

I wonder what else would turn out that way. Drivers licenses for illegal immigrants? (It’s no secret they drive; I’d much rather that they had to follow the same rules I did to get my license.) Loosening up gun laws on non-criminals? (Gotta get both sides in there! I do think Boston and New Hampshire are very different, but I live in a state with incredibly lax gun laws and would be hard-pressed to think of a single gun crime.)

Though I think there’s an interesting lesson in this. For all the political apathy we’re accused of, it seems that there’s an awful lot of Chicken Little FUD (on both sides) about things that turn out to be non-issues to most of us.

After reading about a series of pirate attacks last year—back then an almost laughably bizarre occurrence—I became interested in the concept of modern piracy, something I, like many average citizens, was unaware still went on. I picked up a copy of John Burnett’s Dangerous Waters: Modern Piracy and Terror on the High Seas after hearing him talk on NPR, but didn’t get far into it.

Recent events revived my interest, and I made some headway in the book this weekend. It turns out that piracy has been a major problem for ships in third-world areas, which is problematic since many major international shipping lanes progress right through these areas. No ship is immune, from small sailboats to “VLCCs”: Very Large Crude Carriers, commercial oil tankers rivaling our military’s biggest ships in size. As we learned with the recent hostage situation, pirates tend to be destitute teenagers from the poverty-stricken nations who have little to lose and everything to gain.

This afternoon, I read an interesting observation: some private ships, including cruise ships, are known to employ “heavies,” gun-toting mercenaries, to protect the ship and those onboard. Guns are otherwise uncommon: there are many thorny legal issues, including the need to declare them to customs when docking in a foreign port, at which point they’re seized until you leave again; the fact that pulling a gun on pirates, unless you’re a well-trained marksman, is likely to get you shot; and the fact that, on many of the oil tankers, a single stray round could blow the whole ship up.

So imagine my surprise when I checked out Google News, and saw that an Italian cruise liner off the coast of Somalia actually used its heavies to deter pirates. Besides idle fascination n the escalating pirate wars, I think this is a good thing: if pirates are becoming brazen enough to fire on cruise ships, there’s a much more pressing need for the international community to aggressively put an end to piracy. Piracy is no longer an obscure issue affecting an incredibly small number of commercial ships, but something threatening anyone on a boat in international waters, and the latest escalation is likely to cause an even greater escalation in piracy defenses.

Apparently, a company wrote an application for the iPhone called Baby Shaker. It depicts a crying baby, and you vigorously shake the iPhone to make it stop, at which point its eyes are replaced by X’s.

Apple pulled the application from its store and apologized, saying, “This app is deeply offensive and should not have been approved for distribution on the App Store.”

The Sarah Jane Brain Foundation, however, has had enough, with a spokesperson calling it “the most cynical apology I have ever seen.” They plan to picket Apple stores, calling on them to “mitigate the harm they’ve now caused.”

What I find so interesting is how the Sarah Jane Brain Foundation has had “The PETA Effect” here, at least for me: so vehemently overstating your cause that you steer people to the other side. If I’d seen the application distributed, I’d surely have joined the Sarah Jane Brain Foundation in finding it horrifically offensive. It’s in bad taste and makes light of an abusive practice that kills many babies and leaves even more with permanent injuries and brain damage.

And yet, with them coming across as so overzealous, my “That’s really kind of funny” sense is triggered, just a tiny bit. I guess I find their position so outrageous since:

• I don’t like Apple having sole control of what I can run on my iPhone. Apple pulled the app, which means that, unless I jailbreak my iPhone (voiding my warranty), I can’t have the application. I’m not sure I want this application, but it’s a sore spot with me. The fact that Apple pulled the app just drives home Apple’s exclusive control.
• Apple promptly pulled the app. The “most cynical apology” actually seemed to be a pretty emphatic, “That application was horribly offensive. We screwed up big time in approving it!” from Apple. I’ve definitely heard much more cynical. (I’m sorry you feel that way…)
• The application shows that shaking babies kills them. Sure, it demonstrates it in an awful way, but it’s like showing tapes of babies’ skulls being crushed to lobby against abortion. Isn’t this exactly what the Sarah Jane Brain Foundation should enjoy?
• I think that, and then get the sneaking suspicion that they are loving this, because it’s giving them tremendous publicity. And calling for protests outside Apple stores, weeks after they pulled the application and apologized for it, only furthers that point.

What do you think? Was Apple’s apology (and prompt retraction) of the app good enough? Should Apple have left it up even though it was controversial?

Thinking about a $22,000 lens got me thinking about “real” cameras a bit more. And it occurred to me that Canon is in kind of a weird spot right now.

Their flagship camera has always been the EOS-1. With digital it was the 1D, which was followed by a 1Ds. The s designates that it’s meant for studio work, with a higher resolution but lower framerate. After a while Canon replaced them with the “Mark II” edition of the 1D and 1Ds, and a few years (?) ago, the Mark III edition.

The Mark IIIs were well received. The 1D Mark III supported up to ISO6400 if unlocked, allowing great low-light performance. The 1Ds Mark III is what really got people drooling, though, with a 21-megapixel resolution. I think it was around 10 megapixels that people started saying that resolution wars should really be considered over. 21 megapixels, in the eyes of many, bests medium-format cameras. People shoot for two-page magazine spreads and billboards with lower resolutions.

The awkward sitution comes from the Canon 5D Mark II. The 5D is still a very high-end line, but it’s meant to be second fiddle to the 1D. But the 5D Mark II boasts 21 megapixels, the same as their flagship 1Ds Mark III. It records 1080p video. And what really wins me over is that it gives Nikon’s D3 a run for its money: ISO6400 out of the box, and you can enable “High ISO” support for ISO 12,800 and 25,600, allowing photos to be taken in absurdly low light. It sells for $2,700, less than half of the $7,000 1Ds Mark III.

So it’s high time for a Mark IV series. I haven’t even seen rumors about it yet, which tend to start long before the camera’s released. But here are some of the things I’d really like to see Canon release in a Mark IV edition:

• Higher ISO support with lower noise. I’m not sure many people even imagined ISO6400 in the days of film (though it looks like there may have been such a thing, though it certainly wasn’t sold in Walmart), but the trend has been started. ISO12,800 and ISO25,600 are kind of experimental modes that remain very noisy (grainy). When I’m in the market for a new digital SLR in a few years, I hope it’s got a noise-free ISO25,600, or higher. Consider that increasing sensitivity just twice more would bring “ISO 100K.” Canon and Nikon, it’s a race. You heard it here first. I want the 1D Mark IV to put Canon in the lead, and Nikon to come out with a D4 to try to one-up them, with the end result being a camera that can take photos in dimly-lit rooms without five-figure lenses.
• Get rid of the mechanical shutter. Sample the sensor for the necessary duration. It seems there are design challenges in eliminating the shutter, but it’s really a vestigial organ on a digital camera. This removes a common spot of mechanical failure, and paves the road to higher shutter speeds. I don’t think any camera (possibly excluding ultra-expensive scientific gear) can exceed 1/8000th of a second shutter speeds right now. Accidentally shoot outside at f/1.8 and ISO1600 on a sunny day and tell me it’s not a limit. (Yes, yes, why would you want to do that? Because I needed the shallow depth of field and forgot my camera was cranked to ISO1600. The real question is: why couldn’t the camera handle it?)
• RAM is cheap. Like $10 for a 1GB DIMM. I doubt cameras have DDR2 DIMMs, but why can I only take a couple shots in rapid succession before I have to wait for the camera to write things out to the card? On the flagship model, give us a crazy-huge buffer.
• For the love of God, give us an LCD that we can see when we’re working outside. And while you’re at it, spend the money on a great LCD. Look at an iPhone screen for a while, in fact, and see what 150 dpi looks like.

I think you could say with relative accuracy that there are three main bottlenecks in a computer: CPU, memory, and disk. There are some outliers that people might try to pile in: video card performance, or network throughput if you’re tweaking interrupts on your 10GigE card. But the basic three are pretty universal.

To cut to the chase: I hit disk bottlenecks sometimes, CPU bottlenecks almost never, and RAM bottlenecks all the time. And sometimes high load that looks to be on the CPU is really just I/O wait cycles. But RAM is special: if you have enough RAM, disk throughput becomes less important. At least, redundant disk I/O, which seems to account for a lot of it.

What interests me, though, is that almost everything is RAM starved in my opinion. My laptop has 2GB and I get near the limit fairly often. I’m thinking of trying to take it to 4GB. The jury’s out on whether or not it’ll see more than 3GB, and others complain that 3GB causes you to lose out a bit on speed.

But here’s the thing. I maintain things like a MySQL server with 32GB RAM. It’s not RAM-bound per se: we could switch to a machine with 1GB RAM and MySQL would still run fine. The memory is overwhelmingly configured for various forms of cache. But it’s not enough: there’s still a steady stream of disk activity, and a non-negligible number of queries that have to write temporary tables to disk.

RAM is cheap. It’d cost me about $50 to buy 4GB of RAM for my laptop. The reason RAM stops being cheap is that most motherboards don’t give you enough room. Both of my laptops can only take two DIMMS, which means I need dual 2GB sticks. They’re both based on older 32-bit chipsets, so I can’t exceed 4GB, but if I wanted to, I’d need dual 4GB sticks, and those are expensive. Even on decent servers, it’s hard to find many that give you more than 8 slots, making 32GB hard to exceed.

So what I’d really like to see someone bring to market is a 1U box with as many memory slots as it’s physically possible to fit in. 1U is still tall enough to have standard DIMMs standing up, and most of them are extremely deep. I bet you could fit 256 slots in. Then throw in a compact power supply, a standard LGA775 slot (allowing a quad-core chip), a good Gigabit NIC or four, and an optional FibreChannel card. No hard drives. Maybe a 4GB CompactFlash card if you really want it to have its own storage. Oh, and make sure the motherboard is pretty versatile in terms of RAM requirements and FSB. Oh, and don’t force me to go with ECC. If this were a single database server, it might be worth buying top-notch ECC RAM. But if this were just for caching things, I don’t care. Cache isn’t meant to be permanent, so an error is no big deal.

256 slots, and you could fill it with ultra-cheap 1GB DDR2 DIMMs. (Heck, at work, we have a bag of “useless” 1GB sticks that we pulled out.) You can get ‘em for $10 a pop, meaning 256GB RAM would cost about $2,560. I suspect the system would command a high premium, but really, it’s just $2,560 worth of RAM and a $200 processor. A 2GB DIMM is about twice as much ($20/stick), but $5,000 for half a terabyte of RAM isn’t bad. Though 4GB DIMMs are still considerably more: they’re hard to find for under $100.

I think this would be a slam-dunk product. memcache is pretty popular, and it’s increasingly being used in previously unheard of roles, like a level 2 cache for MySQL. There are also a lot of machines that just need gobs of RAM, whether they’re database servers, virtual machine hosts, or application servers. And tell me a file server (sitting in front of a FibreChannel array) with 256GB RAM for caches and buffers wouldn’t be amazing.

So, someone, hurry up and make the thing. The key is to keep it fairly cheap. Cheaper than buying 4GB DIMMs, at least.

This falls into the category of things that are simple yet I’m always trying to remember. Sometimes in bash I find myself wanting a for loop, not of the usual foreach type (for i in `ls /etc/`; do ...), but a more classical one (a la C-style for var=0; var<100; var++ ... syntax). Here’s the bash way:

for i in {0..99}; do
echo $i
done

The output is a list of numbers from 0 to 99. For a fun example of when you might need this, consider the case of noticing that Postfix isn’t running on a machine and starting it. Your mail client promptly crashes, and the flurry of output in the log indicates that it’s delivering mail as fast as it can, but complaining that some mail has been in the queue for 180 days. “Holy smokes,” you exclaim, stopping Postfix. “Just how much mail is there?” You’re not entirely sure because you aborted your ls after it sat for a good minute with no output. That’s never good. du -sh shows about a gig and a half.

“I’ve got this under control,” you think. rm -f /var/spool/postfix/*. Ta-da! Wait, what’s this “/bin/rm: Argument list too long” business? That error can’t possibly be good. rm very rarely complains. So we tried a smaller delete size, thinking we could do postfix/1*, postfix/2*, etc. Just step through it. That, too, was too much for rm.

So it ended up taking an absurd for i in {0..99}; do rm -f /var/spool/postfix/$i; done to purge the mail. (And that didn’t catch all of it, either; I’m not sure off-hand how to do this in hex.) Each iteration deleted somewhere around 2,000 mail messages, making me think there were something like a half-million messages. (00-FF, times a little under 2,000.)

Lately I’ve felt that things were going pretty well. I was reading a bit of international news and looking at the international reaction to our presence at the G20 summit, for example. Of course not everyone in the world loves us, but I couldn’t help but feel that our presence was a little different than last time. Our President helped get disagreeing parties to agree, and in general seems to have the world eager to work with us. (I don’t really mean this as a condemnation of Bush, nor is it my intention to heap praise on Obama.)

And then I read what conservatives are saying, and it almost seems like we’re looking at two vastly divergent realities. I see a statesman, they see a closet Muslim who was all too eager to bow to an Arab leader and who went out of his way to apologize for being American. I see a fiscal plan inspired by John Maynard Keynes, they see someone deliberately wasting money for his own gain. I see the first black President, they see the first illegal immigrant President. I see a President who came in after Bush’s first round of financial bailouts and pretty much continued the policy, they see a President who nationalized the banks because he’s a Socialist. Oh, and he wants to take everyone’s guns away, and destroy Christianity.

I’d gone a while without reading the “wingnut propaganda,” and in that time period, I’d come to think that things were pretty good. Obama’s approval rating is something like 70%, and the two parties have been known to work with each other a bit lately, even if it’s been far less than I’d like. (And even if it’s been largely Democrat-led, which doesn’t really make for impressive bipartisanship…) And then I realized that there’s a lunatic fringe that seriously believes he’s a Muslim or a Socialist, and became truly worried. Fiscal conservatives and social conservatives may dislike Obama, and I respect their different views. Divergent views, discussed and brought to compromise, truly leave us better off. But there are thousands, if not millions, of Americans who have literally lost touch with reality. They’re like the MIHOPs of the Democrats.

I also want to caution that when I use terms like “neocon” and “wingnut,” I mean them more literally, not as terms to refer to all Republicans. Similarly, I respect Republicans and hate the artifically-created divide between the parties. What I’m complaining about is the wingnut Republicans who use utter lies to advance their own causes. There are Democrats who do the same, surely, but with Democrats leading Congress and the White House, those people aren’t noteworthy right now.

Anyway, two things have interested me lately. Besides the thousands of dead Americans, one thing that always bothered me about the Iraq War was the exorbitant cost. If the money were spent domestically, it could have gone an amazing distance. The military takes up something like 50% of our spending. So I’m waiting to see what the wing-nut faction of Republicans says. They’ve spent weeks protesting Obama’s Socialist spending. But the Democrats have long complained that the Iraq War is too expensive, and Republicans have argued that not giving the military a blank check amounts of waving the white flag of surrender. So I’m curious where this will go, because it could leave Republicans in an awkward state either way. Hopefully it will just be passed and nasty politics will be left out of it.

But then I was reading this article about how Obama may be looking to get the ball rolling on immigration reform. And the general description of his plan seems to amount to increasing border patrol and cracking down on illegal immigration. During the campaign trail one of the things he discussed was a path to citizenship, but with a pretty steep burden: you’d have to learn English, pay back taxes for as long as you’ve been in the country, pay a fine, and only then would you “get in line, behind everyone who came here legally” to become a citizen. Of course, that was something discussed during the campaign trail. The “official” Administration hasn’t even released a plan yet, but has merely made mention of strengthening border control, and the article is little more than speculation.

Yet some Republican activists have already denounced Obama’s (currently non-existent) plan as “dangerous” and “amnesty.” Seriously.

I’ve always thought that safety and freedom were somewhat incompatible goals. You can have a very safe society, but next to no rights. (A police state.) Or you can go the opposite way, and have a very free society, one in which no one can stop me from detonating nuclear bombs. But there’s a middle ground, I think, where we have a good deal of freedom, but are also pretty safe.

I’ve always thought that airports were a good example of starting to give up too much freedom to get a little safety. While I’m certainly not eager to give up safety, I could do with a slightly less-strict set of policies in airports.

But here’s an even better example. A town in Arkansas is drawing criticism for what seems to be a non-stop curfew, in which anyone on the streets is stopped and investigated. (It’s not terribly clear to me what’s meant by the curfew, though: I think of a curfew as it being illegal to be in public during a certain time, so a 24/7 curfew would basically mean you couldn’t leave your house. And they kind of suggest that, but it seems people are also going on with their daily lives?)

Is it safe? Very much so, it seems. Violence is way down, and they’ve made lots of arrests. But would you want to live there, where walking to the store was cause for the police to detain you? (The telling quote is, “The citizens deserve peace, that some infringement on constitutional rights is OK…”)

Have you heard about McCain’s new ad? It’s on his main page, though there seems to be no way to direct-link to it. Somewhat bizarrely, it starts off with video of throngs of cheering Obama supporters, and calls him the biggest celebrity in the world, flashing images of Paris Hilton and Britney Spears. (By the way, Paris Hilton apparently had no knowledge she was being used in the ad; using one’s likeness in commercials is generally illegal, though I have no clue if political campaigns are exempted.)

It then goes on to say, “But is he ready to lead?,” before attacking Obama for opposing offshore drilling (I’ll save that rant for another time), but, more significantly, talks about his plan to raise taxes on electricity. Yipes, that’s bad! Raising taxes now? On electricity?!

There’s one problem, though. It’s not at all true.

Newsweek has a good article explaining where McCain’s campaign got the quote about Obama wanting to raise taxes on electricity. In an interview, he was asked, “Have you considered other funding sources, say taxing emerging energy forms, for example, say a penny per kilowatt hour on wind energy?” You can read the quote for yourself, but his answer was essentially that taxing renewable energy was an awful idea; taxing ‘dirty’ energy would make more sense, but even that isn’t the real solution to funding education. And yet, if you quote just one sentence from the middle with no context, you can make it seem like he’s saying that we need to raise taxes on electricity. Except that he was making the exact opposite point.

When Obama’s campaign criticized the ad as baseless FUD, McCain then went on to accuse Obama of “playing the race card.”