Measure All the Things

Those of you who know me well will probably know that I kind of have a thing for graphs. More generally, I find monitoring, trending, and logging to be invaluable. No sane sysadmin would ever set up a server without those things.

But why doesn’t my house support this? Over the winter, I noticed that my furnace was periodically throwing an error code. It worked fine, but would eventually display an error code, shut down, and start back up again. I have no idea what causes it or how often it happens. Since the error code’s description isn’t anything hazardous-sounding, and since the system operates fine on the whole, I haven’t yet considered paying someone to come out and look at it.

Meanwhile, my problems aren’t limited to heat. My air conditioner has been fraught with problems. When the technician comes, he’ll attach a set of gauges to the lines to measure pressure. A more thorough technician might also attach temperature clamps to see the exact temperature going in and out, which provides a sanity check and allows him to calculate superheat and subcooling. It’s also not uncommon to measure current of various components and see if they are within normal parameters. Of course, my system, being designed by buffoons, seems to make this stuff really hard — a decent number of measurements require disassembling the thing.

And as all of this goes on, I’ve been tempted to install this Brultech ECM1240 meter, which measures current on each circuit of your home’s electrical panel. I don’t need it, but I really have no concept of how much electricity I’m using (beyond the monthly bill), nor what uses the most.

But all of this leaves me frustrated. When my furnace hit an error, why couldn’t it send an SNMP trap, fire a syslog message, or send me an email? (Or, for that matter, send an alert to my HVAC company, which could then log in and look at electronic diagnostics, having a good idea what the problem was before anyone came out?)

When they come out to fix my AC, why do they have to bring their own gauges? Why can’t the tech just pull out an iPad and read the values over Bluetooth, getting not just pressure but a wealth of other information that the AC is already tracking for its internal operations? And, when he says, “It seems like you’ve got a leak,” why can’t we pull up a graph and see the pressure decreasing over the past week? And for that matter, why didn’t it just send me an email alerting me that the pressure had fallen below the expected range, before the thing got so low that it didn’t work?

And when I really want to troubleshoot more, why can’t I just set things to log in more verbose mode? Why doesn’t my thermostat send an INFO event whenever it kicks a zone on or off, which can just live in a ring buffer that’s generally ignored until something goes wrong? And before my smoke detector sounds the alarm, can’t it send me a warning that it’s detecting light smoke and will go off soon? (As an aside, the concept of a ‘pre-alarm’ is not a new one.) And when it does randomly sound an alarm at 4:30am, why can’t it send me a text message telling me what detector has fired, so I don’t have to run around in a panic before realizing that it’s the smoke detector in my bedroom, where there is clearly no smoke or fire? (And, for that matter, why can’t I reply “stfu” to stop it from sounding?)

What frustrates me about this is that it’s like I’m describing a futuristic, almost science-fiction world. SNMP is almost as old as I am. Ethernet has been around for more than 30 years. syslog has been around for about as long. Everything I’ve described could have been fully implemented in 1990. People have been talking about “smart homes” before then. More than two decades later, when everything has computerized onboard diagnostics already and boards that can do Ethernet, syslog, SNMP, and an embedded webserver are incredibly cheap, this is still a pipe dream. Why?!

Quotes of the Day

Two great quotes from recent news, providing a much-needed infusion of common sense and sanity:

“When anyone, not least a member of Congress, launches specious and degrading attacks against fellow Americans on the basis of nothing more than fear of who they are and ignorance of what they stand for, it defames the spirit of our nation, and we all grow poorer because of it.”

— John McCain, condemning Michelle Bachmann’s assertion that Hillary Clinton’s top aide was a spy for the Muslim Brotherhood.

“As a Catholic I believe in this religion, there are amazing things about this religion, amazing moral teachings that I do believe in, but I also have to think about how we keep women alive. I believe in not letting women die, I believe in not letting babies die, and to me that’s more important than arguing about what method of contraception [is right].”

— Melinda Gates, pledging $560 million to provide contraceptives in the developing world, despite the Vatican’s opposition.

Flip

Although it was quite orthogonal to what I was looking at, Flip Your Classroom just came up as a related item on Amazon. It’s one of those rare Amazon items with a solid 5-star rating, and, despite not being a teacher and having absolutely no intention of becoming one, I’m so intrigued by this bit of the book’s summary that I feel compelled to buy and read it:

It started with a simple observation: students need their teachers present to answer questions or to provide help if they get stuck on an assignment; they don’t need their teachers present to listen to a lecture or review content. From there, Jonathan Bergmann and Aaron Sams began the flipped classroom–students watched recorded lectures for homework and completed their assignments, labs, and tests in class with their teacher available.

Energy

Every now and then I start reading something on Wikipedia, and end up getting pulled into a long series of digressions wholly removed from what I set out to read about. But it’s invariably fascinating.

Take, for instance, solar power. I’ve read before that solar panels are really quite inefficient. At its peak, the energy from the sun reaching Earth is about 1kW per square meter, but solar panels tend to convert only about 10-15% of that into power — 100 to 150 Watts per square meter. That seems almost impractical. And then there’s the high capital cost upfront; Wikipedia suggests it takes about a decade before you’re recouped your costs. So I had, perhaps naively, assumed that solar power just wasn’t workable.

But it turns out that the sun actually provides massively more power than we need. This diagram on Wikipedia shows the hypothetical placement of large solar farms across the world in such a way that would meet all of the world’s power demands, taking into account average cloud cover, and assuming only 8% efficiency. (Recall from above that it’s actually a little bit better than that.) Granted, there are all sorts of practical problems with this plan (for example: the people who currently live under the black dots, or how to export electricity from Algeria to the United States in an efficient manner), but it’s kind of intriguing to know that it’s actually possible.

Of course, the output of solar panels doesn’t align with our usage of power. (For example: sometimes people want to use electricity when it’s dark outside.) There are various means of storing energy, and it turns out that batteries really don’t scale to massive needs very well. An overwhelming majority of “bulk energy storage” is in the form of pumped-storage hydroelectricity — when you have a surplus of power, use it to pump water up a hill. When you have a shortage of electricity, let gravity run the water down the hill and run a hydroelectric generator. The world’s largest installation of such a system is the Bath County Pumped Storage System, which, when releasing energy, is capable of an astonishing 13.5 million gallons per minute water flow, and which has a capacity of just over 3,000 Megawatts. The efficiency of such a system is unclear to me, but it operates at a net loss.

Of course, no energy-storage system is going to operate at a net gain. The goal is to store power when you have a surplus of power being generated, and release it when you have a deficit. But of course you want to be as efficient as you can in doing this, and there are a few methods beyond pumping water up and down a hill.

One method that intrigues me is that of using flywheels — spin up a large rotor inside a vacuum-sealed enclosure. The latest designs suspend the rotor on magnets, further increasing efficiency. The net efficiency is apparently close to 90%. (Another interesting aspect of flywheels is that they’re capable of building up a massive charge over time and then discharing it near-instantaneously, leading to their use in niche fields where you need a sudden surge of current that would wreak havoc on the power grid — such as testing a million-Amp circuit breaker.)

But more interesting still is the idea of using superconducting magnets. Superconductivity itself fascinates me — as you approach an absolute zero temperature, you reach a point of superconductivity where resistance becomes exactly zero. It’s possible to accomplish with liquid nitrogen. While superconductivity has all sorts of interesting applications, one is that you can apply a charge to a superconductor and have it persist without discharge indefinitely. Seemingly the only inefficiency is in the AC/DC conversion, plus the overhead of keeping the superconductor in a cryogenic state, but it sounds like net efficiency is around 95%. (Though they apparently don’t scale up very well right now.) As an aside, although it sounds like superconductivity is a sort of bleeding-edge science, the concept was discovered in 1911, making it more than 100 years old.

And even though the sun can apparently meet our energy needs many times over and we can store the surplus energy from sunny days with very high efficiency, there’s always another source of power: garbage. Using plasma to vaporize garbage appears viable. And while conventional incineration is a tremendous source of dangerous emissions, vaporizing it with plasma atomizes it, and anything remaining is turned into a glass-like pulp. (This technique has apparently seen prior applications in cleaning up nuclear waste, leaving behind an inert glass-like substance.) And while vaporizing garbage with plasma consumes a tremendous amount of energy, it also releases a tremendous amount of energy, with the potential to be a net producer of energy.

Direct Democracy

As luck would have it, I’ve had multiple discussions lately that centered around the concept of direct democracy.

Fundamentally, the power of the United States of America rests with we, the people. (Forgive the grammatical inaccuracy. I was making a point.) But in practice, if you stopped someone on the street and asked who runs America, I’m pretty sure no one would say “you and me, buddy!” You’d get answers like, “Obama,” “Congress,” “Special interests,” or the political parties.

As a matter of practicality and convenience, since we couldn’t have all 300 million of us gather in one place to talk, and since we have busy lives to lead, we have appointed some delegates. We pay them a generous salary to do our bidding, and call these employees of ours “Congress.” But how many people believe Congress is actually representing their interests? The question is not rhetorical. The answer is 9%.

We hire people to represent us in Congress because we don’t have the inclination to do it ourselves, and because it’s not practical for so many people to meet. But I content that both of these premises are wrong.

People talk about politics all the time. We read about politics in the newspaper, and we forward each other political emails. We monitor elections in other states to see if the people who share our political beliefs are elected. When a terrible piece of legislation is put forth, millions of people reach out to try to stop it, but feel mildly powerless to effect change. People are involved in their government, and feel like they don’t have enough involvement.

As for 300 million people meeting, the Internet has changed this. True, it’s not remotely feasible for 300 million people to gather in a meeting hall. But having 300 million people collaborate on something online? Sites like Facebook and Google handle massively more users than we have citizens. And sites like Wikipedia and Github prove that people can come together and build amazing things.

But when I bring this concept up, there are two problems that everyone I’ve mentioned it to have raised. I instinctively agree with them, but, thinking more about, I’m not sure either is worse than status quo.

The first issue is that hackers will have a field day with it. And there’s some truth here. Electronic voting machines are ridiculously insecure and error-prone, to the point that a huge number of geeks actively oppose the use of electronic voting machines. It’s one thing to be able to break into my Facebook account and post spam, but it’s something entirely different to hijack my vote. Security would have to be a key consideration central to the design of such a system, but that’s a design requirement, not a reason to write the whole thing off as infeasible. But second, opposition on the basis of  security assumes that our system is presently secure. When you do vote for candidates, voting fraud is a known issue, although it’s not nearly as rampant as in other countries. But how often do you even get to vote on the issues that affect you? You write to your Congressman, who reads your email, but who also hears a lot from lobbyists who can wine-and-dine him in a way you cannot. So I have to wonder if, even if the system were wildly insecure, it would actually give users less of a voice than they have now. I’m inclined to think that it would still be an improvement.

The second issue is more surprising: do we actually trust people with a vote? Read the comments on any major news site some time, or worse, the comments on Youtube, and you will be horrified at the prospect of those people voting. And then consider all the falsehoods and fake emails that get sent around to try to scare people into voting for their preferred candidate. And it turns out that this question of whether everyone should be able to vote on everything goes all the way back to the American Revolution, and the general answer was “No,” which is why we have a representative democracy. It’s not just that the Founders saw practical issues: it’s that they didn’t trust everyone!

But I contend that this, too, is wrong. (Sorry, Alexander Hamilton.) There are, admittedly, a ton of people whose ability to vote terrifies me, and I completely agree with many of the observations about the pitfalls of allowing everyone to vote on issues. But I reject the notion that allowing us to pick representatives to make decisions for us somehow solves the problems of tyranny of the majority that the Founders thought were inherent to a direct democracy. And the whole model is based on the offensive notion that our elected officials know better than we do.

To be sure, this isn’t a simple problem. Given the pace of anything involving the government — and the fact that incumbents (and special interest groups) have every reason in the world to oppose being replaced by a website — I can’t imagine that this could be put in place in less than a decade. But that’s all the more reason to start thinking about it now.

Investments: Good, Great, and Excellent

I have very little money invested in either account, but this isn’t actually about money for those of you who are just skimming the article.

401(k) plan over the past year: A good investment. (Sort of.)

 

Actively-traded brokerage account I opened with a small amount of savings as an experiment after realizing my bank was paying me 0.25% interest on that money, and after borrowing some concepts from Joel Greenblatt but only investing in companies with excellent analyst recommendations when they dipped below their normal range without substantial cause. (Axis is chopped to not show off my pitiful balance, but the rate of return is approximately 30%.) Overall: Great investment.

Key Lime Pie, frozen, bought at supermarket and stored in freezer for whenever I want a slice, even if it’s not a tenth as good as something you’d find in a gourmet restaurant or in Key West: Excellent investment.

Wireless Network Info Card

Like any good geek, I locked down my wireless network a bit after I set it up. I’m running WPA2, and the password/key was generated with pwqgen, for something sufficiently secure but semi-memorable. I disabled SSID broadcast. My router automatically generated a weird name, which I presume to be two random dictionary words mashed together. More for congestion-avoidance than security, I also run the router on 5 GHz instead of 2.4 GHz, which limits me to 802.11a and 802.11n. This seems to work with most everything (except for my HTC Inspire cell phone).

The problem is that when people come over, it’s really tiring to hand them a sheet with the network name and password/key on it, and still have to explain:

  • My network isn’t any of the ones that pop up on your wireless device, because I turned SSID Broadcast off. You have to manually add it.
  • It’s WPA2, not WEP. It’s also Personal, not Enterprise, so there’s no username prompt.
  • The name is weird, because it was auto-generated. Stop asking what it means; it means nothing.
  • It’s only on 5 GHz.

So I decided I wanted something more than a sheet of notebook paper, but also something that people could “get” pretty quickly. I grabbed a photo of the iOS network setup, guessed that Helvetica Neue was a close-enough match for the font, and did a little Photoshop manipulation, ending up with this:

(These values are randomized and not my actual credentials.)

My plan (once I get my printer set up and working) is to print a couple copies of this on thick cardstock like an index card and keep them handy for when guests want to use my network.

Photoshop template here [720KB, .psd], if you’d like.

Stumped by (13)Permission denied: file permissions deny server access ?

I was just pretty stumped by the following Apache error, in /var/log/httpd/error_log:

[Thu Dec 08 21:53:28 2011] [error] [client 192.168.1.2] (13)Permission denied: file permissions deny server access: /var/www/html/aml/i
ndex.html

The error implies it’s a filesystem permissions issue, and the Apache site explains that it’s almost always related to filesystem permissions, not Apache configuration. But this was exasperating, because the file ownership was apache.apache, and index.html was chmod 755. Apache could absolutely access it. But then the Apache docs made a passing mention that sometimes it was related to SELinux errors.

Sure enough, that’s exactly my problem:

[matt@bos aml]$ ls -alZ 
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 .
drwxr-xr-x. root   root   system_u:object_r:httpd_sys_content_t:s0 ..
-rw-r--r--. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 .htaccess
drwxr-xr-x. apache apache unconfined_u:object_r:user_home_t:s0 images
-rw-r--r--. apache apache unconfined_u:object_r:user_home_t:s0 index.html
drwxr-xr-x. apache apache unconfined_u:object_r:user_home_t:s0 stylesheets

Note that the directory, and the .htaccess file I touched, have the httpd_sys_content label, which is appropriate here. But the other files show user_home_t. This apparently precludes Apache from being able to see them.

Where this went wrong is that, to get them onto the server, I scp’ed them from my desktop to my home directory on the webserver, and then cp’ed them to /var/www/html. In doing so, they picked up the user_home_t label since they were in my home directory.

A quick sudo restorecon -r . got everything sorted out. There may be a more elegant way, but this worked for me.