Category Archives: Ideas

Idea

Posted on by
1

Why isn’t there a really good “network appliance” as a network gateway? You can get a low-end firewall/router, or you can build your own machine.

Setting up OpenBSD is no walk in the park, though. I want to build an “appliance” based on OpenBSD, and give it a nice spiffy web GUI. You buy the box, plug one side into your switch and one side into your cable modem or whatnot, and spend ten minutes in a web browser fine-tuning it. I was really fond of the appearance of the Cobalt Qube, although it could be made much smaller. And throw a nice LCD on the front with status. You can run a very low-power CPU, something like the one powering these. It really doesn’t need more than 512MB RAM, but give it a small solid-state drive. And a pair of Gigabit cards, not just for the speed, but because GigE cards usually are much higher-quality. In building routers, the quality of your card determines how hard the CPU has to work.

There’s so much that a router can do. You can run a transparent caching proxy, a caching DNS server, priority-based queuing of outgoing traffic (such as prioritizing ACKs so downloads don’t suffer because of uploads, or giving priority to time-sensitive materials such as games), NAT, an internal DHCP server, and, of course, a killer firewall. You can also generate great graphs of things such as bandwidth use, blocked packets, packet loss, latency…You can regulate network access per-IP or per-MAC, and do any sort of filtering you wanted. It could also easily integrate with a wireless network (maybe throw a wireless card in, too!), serving as an access point and enabling features like permitting only certain MACs to connect, requiring authentication, or letting anyone in but requiring that they sign up in some form (a captive portal). And I really don’t understand why worms and viruses spread so well. It’s trivial to block most of them at the network level if you really monitor incoming traffic.

I’m frankly kind of surprised that nothing of this level exists. I think there’s a definite market for quality routers. A $19 router does the job okay, but once you start to max out your connection, you’ll really notice the difference! A good router starts prioritizing traffic, so your ssh connection doesn’t drop and your game doesn’t lag out, but your webpages might load a little slower. An average router doesn’t do anything in particular and just starts dropping packets all over the place, leaving no one better off. (And a really bad router–our old one–seems to deal with a fully-saturated line not by dropping excess packets or using priority queueing, but by reboot itself, leaving everyone worse off… I think this may have had to do with the duct tape.)

Geek

Posted on by
Reply

We’ve been having a lot of intermittent network problems at home. Periodically, our Internet cuts out. At first I assumed it was our ISP–it’s no longer Adelphia (run by pharmacists), though–but subsequent research indicated that it wasn’t our ISP’s fault: our router was going down.

My dad set it all up, so I wasn’t too sure how things went. I was pretty confident that we were just using a generic store-bought broadband router, though, so I found it strange that it would be drifting in and out. It turns out that I overlooked something about the router: it’s being held together with duct tape.

I’d already been intrigued by OpenBSD’s pf, so this seemed like a sign! I commissioned an old desktop system, loaded OpenBSD up on it, and went to work configuring it. OpenBSD was just more different from Linux than I expected. It asks you if you want to let OpenBSD use the whole hard drive. I said yes, and thought, “Wow, this is just as easy as Ubuntu!” But it turns out that this was just the first stage. After this, you have to set “disk labels,” which are sort of like partitions but ambiguously different. The syntax is obscure, the purpose is obscure, and so forth. Then I had to configure the network. NICs are named by the drivers they use, so instead of eth0 and eth1 (for Ethernet), I have rl0 (Realtek) and dc0 (who knows).

I was also extremely confused trying to set up routing. Long-term, it was going to be the router, but short-term, it needs to know about our existing router so that it can connect and download the requisite packages.

So I finally got it all set up. I also installed MySQL (unnecessarily, it turns out), Apache, and PFW, a web-based configuration tool for pf. I ended up not using PFW, because my understanding of pf is so bad that I’m basically relegated to copying-and-pasting rules from websites into the configuration file.

Even using pf is confusing. It’s called pf, but typing “pf” at the command line doesn’t do anything. It turns out that you control it with a tool called “pfctl.” You can do pfctl -e to enable pf, and pfctl -d to disable it.

As I tried to tweak the firewall/routing rules, I’d periodically “restart” pf by disabling and then re-enabling it. I wasn’t sure if it read the rules “live” or if a restart was needed. It turns out… neither! The rules are stored in memory, but restarting pf doesn’t flush the rules. You need to pass pf some more arguments to tell it to flush the cache and read them anew from its configuration file.

After a few more hours of work, I thought it was all set up. Both NICs were configured, the external one to get an IP over DHCP, and the internal one with a low fixed IP. I had a complex set of rules, doing NAT, filtering traffic, and using HFSC for prioritized queueing. (HFSC seems completely undocumented, by the way. I took my tips from random websites.) It seemed very impressive: I prioritized ACKs so that downloads wouldn’t suffer if our outbound link was saturated. (Aside: it really doesn’t make sense to do queueing on incoming traffic, since the bottleneck is our Internet link, not our 100 Mbps LAN.)  I also afforded DNS, ssh, and video game traffic high priorities, but allocated them a lower percentage of traffic. I even figured out the default BitTorrent ports and gave them exceptionally low priority: if our line is fully saturated, the last thing I care about is sharing unnecessary data with other people.

And there are other neat features. It “scrubs” incoming connections, reassembling fragmented packets and just eliminating crap that doesn’t make sense. It catches egregious “spoofing” attempts and discards them.

I hooked up the second LAN connection to test it out, rebooted, and… waited.

It never came up. Well, it did come up. The computer’s running fine. Both network cards show up with the switch. Doing an nmap probe of our LAN, I see one strange entry. It’s actually pretty mysterious: it has no open ports, and attempting to ssh into it just sits there: it doesn’t send a connection refused, but completely ignores the incoming packets, leaving my poor ssh client sitting there waiting for a reply, having no clue what’s going on.

In a nutshell, it seems that I just built a firewall/router that’s so secure that I can only find one of its two cards on the network, and I can’t even try to log into it. Let’s see you hack that! Of course, this does have some issues. For example, I can’t use it.

I haven’t lost hope yet: I have a keyboard and monitor so I can log in on the console and try to do some tweaking there. (You can’t firewall off the keyboard.) It’s just not very encouraging to think, “Alright, let’s reboot and make sure it works as flawlessly as I think it will” and then have the darned thing not even show up on the network.

An Uncontrollable Urge

Posted on by
1

A few years ago Andy and I ran a hosting company. It never got that far, but it was fun, and also a learning experience.  Today I’m finding that I can’t get the idea of starting it again out of my head. The problem is that, this time, I’d want to start it big.

There are a bunch of technologies that I find downright exciting:

  • Old racks full of blade servers are hitting the used market. And by “old” I mean dual 2-3 GHz Xeons, a gig or two of RAM, and hard drives that still rival what hosts are renting in dedicated servers. I’d probably want to put in new drives, but the machines are cheap and they’re plentiful.
  • Boston has a number of good data centers, and all the big Tier 1 providers are here. That there seem to be no well-known hosting companies out here is frankly kind of surprising. You have no idea how badly I want to pick up a couple racks in a colocation facility, and pull in a couple 100 Mbps lines.
  • cPanel looks like it’s matured a lot since I last used it, and it has some good third-party stuff such as script installers. It looks like it remains the number one choice in virtual hosting.
  • Xen is downright exciting. It permits splitting a physical host into multiple virtual machines. With the advent of chips with hardware virtualization support from both AMD and Intel, it now runs with very little overhead. It used to require extensive modifications to the “guest” OS, so that only modified versions of Linux worked. With newer processors, though, you’re able to run machines without them having to know they’re in a virtual machine, opening up options. You can run Windows now. The virtual dedicated server / virtual private server market is growing. (Xen also supports moving hosts between physical servers, which has a lot of nice applications, too!)
  • OpenBSD’s firewall, pf, continues to intrigue me for its power. I just found PFW, a really spiffy web GUI for managing pf. Not only does it do basic firewall stuff, but it’s got support for prioritization of traffic / QoS, and for load balancing. I’m probably just scratching the surface.
  • I’ve spent years honing my admin skills and improving server performance. Improved performance on a shared server, of course, means more clients per server, or more money.

I’m wholly convinced I should start a Boston hosting company. I just need $100,000 capital or so. (Santa, do you read my blog? Do you fund businesses? I’ll give you partial equity.)

Das Klunker

Posted on by
Reply

In the past few days, we’ve spent way more time shoveling snow than is reasonable. We have a relatively long driveway, and it’s a major pain to shovel it. We have a snowblower, but it’s really not all it’s cracked up to be. And last time I was in the garage half of the engine appeared to be removed.

While I think it’s more of a pipe dream than anything, my dad and I have been kicking around the idea of picking up an old junker of a truck. It should run and pass inspection (or be reasonably easy and cheap to fix), but some rust and crappiness is okay. Something like this. And then we’d throw a plow on the front. It’d be useful for our weekly runs to the dump, and, of course, for plowing. (Plus, there’s a lot of “cleanup” we’d like to do–there’s probably about two feet of snow on the side of the road that could be pushed back, and the town did a terrible job plowing the entrance to our street. And there are huge snowbanks, something we’d love to push down a bit.

It turns out that plows are expensive, and good, running trucks are also way more than I thought. From having seen junkers on the side of the road in the past, I was fairly convinced we could do this for under $500. It’s now looking like $2,000 is a better price range.

Or we could just get a just-broken-in Lincoln (Presidential Series!!) for a mere $500. (Step one: remove garbage from floor. Step two: try to clean up the leather a bit. Step three: resell.) Actually, I’d love to try to take my ‘radio arbitrage’ skills and apply them to autos, except that having an extra radio is well and good, but having four broken-down cars in my parents’ driveway might not be as well received. And, you know, the fact that they’re a lot more expensive and thus a lot more risky.

Qqueue!

Posted on by
Reply

So I’m a huge fan of Ask Metafilter. The basic premise is simple: you ask a question and lots of people answer. But Ask MeFi rocks because they maintain high standards. So you actually get really good answers. It costs $5 to join, which is done to pay for the servers but, frankly, seems like a good way for keeping crap out, too.  You’re allowed one question a week, so I try to make it good. But oftentimes, I put it off for several weeks for want of something worthy of using up my question.

So I started a list. And I figured I’d allowed voting and comments. And before I knew it, I had this monstrosity. It was actually extraordinarily simple to code, too. I hope to add better questions over time: these are the ones that were on my mind at the time. You can vote (the + and – buttons), and leave comments. Feel free to do so. (I’m not taking question ideas: get your own account if that’s what you want!)

Saving the Auto Industry

Posted on by
2

My whole family drives Toyotas. We love America and all, but we want good, solid cars. The U.S. is, understandably, concerned about how much oil we’re using. So we’re trying for a requirement that, by 2020, all cars sold get 35mpg at a minimum. Of course, the car companies are complaining that this is going to be incredibly difficult to do.

Two comments:

  • This is utter BS. My mom gets 50 mpg with her Prius. Honda did it in 1987.
  • Why does the government need to get involved? The way I think it should be working is that we say, “$3 a gallon for gas is ridiculous! I want a car that gets better gas mileage!” We stop buying cars that get horrible gas mileage, and, consequentially, Detroit stops making cars that get horrible gas mileage because no one is buying them. It costs me $40 every time I fill up. I wince every single time.

I found this video online. I’m not going to lie: it’s dry, and 20 minutes long. I was kind of proud to follow him most of the time as he talks about internal rates of returns and demand pull and the like. He makes some extremely obscure references, and even now, I’m not sure what he was talking about with oil at $12 a barrel.

And yet, despite it being presented in a technical, academic manner to an audience that’s definitely not normal people, he makes some points that are really, really, really worth hearing. One of the simplest ones: efficient cars are going to be made, the question is who’s going to make them. And, at least right now, it’s not us. (And it really boggles my mind, frankly. Ford makes one hybrid: the Ford Escape Hybrid. 34mpg on an SUV is impressive (I get 20-22). But what the heck market are they appealing to? They manage to completely dilute the effects of a hybrid engine by putting it in an SUV.)

GM developed a “concept car” 16 years ago that, as I recall, got close to 100 miles a gallon. Where is it?

Besides oil, another huge problem we’re facing is a ridiculously huge trade deficit. If we could make cars good enough that we wouldn’t have to keep importing cars, we could certainly help.

He presents some amazing statistics, too. 87% of the energy from fuel used in cars is utterly wasted. Only 6% of the total energy actually moves the car. (And when you figure in that the car weights significantly more than the passengers and luggage, he says that less than 1% actually moves the passengers.)

He says the solution is to lighten the car. I cringed for a minute. Lighter cars, especially on today’s roads, are asking for disaster. You can go drive your 500 pound car, and I’m sorry if I kill you when you crash into my SUV.

But it turns out that this is somewhat wrong. He showed a picture of a McLaren SLR (a $400,000+ car) that was made out of carbon fiber. It’s very light. Some idiot T-boned the car. Their car was totaled. The McLaren driver had to buff out a scratch in the paint. He suggested that, if you were to smash the car head-first into a brick wall, about 25 pounds of carbon fiber is all it would take to absorb the impact and let you walk away unharmed.

He goes on to call heavy cars “hostile cars,” and really, he’s got an excellent point. We’re making heavy cars solely for safety with other cars. But we can increase fuel efficiency, maintain (or increase!) driver safety, and decrease risk to other motorists by simply changing materials.

Oh, and one final point he makes that I thought was interesting: we think of OPEC as a cartel that has tons of power. In actuality, our power of demand far outweighs their supplier power, and we have the power in the equation. Except that we can’t stop buying oil. Years ago we saw a lull in demand, and basically gave OPEC the bird. He suggests doing it again.

I didn’t expect to watch the whole video, which is 20 minutes long. But before I knew it I was done. And it’s pretty thought-provoking.

Why I’d Go Nikon

Posted on by
Reply

Andrew’s biased me. I’m a Canon fan. I own a Canon body, and now, two Canon-mount lenses. And this brings in switching costs: the lenses would be useless to me if I had a Canon. And, while I think it’s mostly irrational, I’ve come to love everything about Canon cameras and see any difference as a flaw in Nikons.

But I’m still excited about the Nikon D3. And it turns out that I’m far from the only one. The D3 has a ton of people anticipating its release. And even at 5 grand, I wouldn’t be surprised if they’re sold out at first. I don’t have that much to spend on a camera, but if I were a serious photographer, I’d have pre-ordered mine already.  Why?

  • Higher ISOs mean you can get shots that you otherwise couldn’t. Or that you can buy cheaper (and lighter) lenses and still get good shots. Everything in photography is a trade-off: to increase shutter speed, you need to either raise the ISO (which raises grain), or use a wider aperture (which usually hits hard limits: your lens is only so good, and you pay through the nose for faster ones). Increases in usable ISO, though, come “free”–if you can suddenly take clean shots at ISO6400, as you apparently can with the D3, you can get shots that, frankly, were impossible on other cameras.
  • Higher ISOs can mean increased savings. To get really good shots when I can’t shoot above ISO1600 (or ISO800 if I want clean shots), I pretty much have to buy a faster lens. Pros have tens of thousands in high-end lenses for just this reason. They can get the shots I can’t. Suddenly, at ISO6400, I’d be on par with them.
  • A lot of cameras are using “cropped” sensors… The sensor is smaller than 35mm film, so only the center of the image coming through the lens falls on the sensor, effectively cropping the image. This is beneficial if you’re using telephoto lenses, as it’s essentially a “bonus” zoom. (A 200mm lens on my camera is equivalent to a 320mm lens on a full-frame camera.) But for people who shoot at the wide end, it’s a major pain. The crop gave rise to things like Sigma’s 10-20mm lens, which is ridiculously wide. The reason is that, on a 1.6x crop sensor, it’s 16mm equivalent at the wide end: right on par with existing lenses. A lot of lenses are being built just for these cropped sensors, which permits them to be lighter and cheaper. But people still prefer the full-frame sensors, so now there are two types of lenses floating out there. Nikon nailed it here: their camera will work with both. If you mount a lens for ‘cropped’ cameras, it’ll only use part of the sensor. If you mount a full-frame lens, it uses the whole frame.
  • They built a longer-life shutter. Bravo. (Actually, Canon did too…)
  • They improved the LCD to over 900,000 pixels. One thing that drives me nuts on the 10D is that the image is tiny and low-resolution. You have to spend time zooming in to see if it came out alright. And when you’re shooting live action, this means missing a ton of shots. So you shoot blindly, and then realize that the whole thing came out unusable.
  • They have a built-in guide, so you don’t have to carry the manual around. Again, brilliant! The menu also looks a little bit less like it was made in 1982.
  • When I talk about high ISOs, 6400 is just their ‘normal’ upper. As with most cameras, you can enable “Expanded ISO” mode, which gives you some more settings, with the catch that they’re somewhat noisy. But you can shoot at ISO25,600. This is just obscene, and I’m fairly certain that, until Nikon came out with this, no one had ever even thought about a camera being this fast.
  • They kept up a high shutter speed… Between 9 and 11 frames per second, in fact.

Something tells me that the folks at Canon are scrambling to develop a sensor this good.. I hope they are. Because I hate those stupid circular viewfinders on Nikon cameras.

Aside: I really hope the folks at Canon are also scrambling to develop a camera that ditches the shutter… I’m still at a loss to explain why it’s even in a digital camera.

Aside: Maybe they can steal my ideas and include a useful integrated WiFi chip… Or a built-in intervalometer. That’d be trivial to implement?

memcached

Posted on by
3

On my continuing series of me poking around at ways to improve performance…

I accidentally stumbled across something on memcached. The classic example is LiveJournal (which, incidentally, created memcached for their needs). It’s extraordinarily database-intensive, and spread across dozens of servers. For what they were doing, generating HTML pages didn’t make sense that often. So it does something creative: it creates a cache (in the form of a hash table) that works across a network. You might have 2GB of RAM to spare on your database server (actually, you shouldn’t?) and 1GB RAM you could use on each of 6 nodes. Viola, 8 GB of cache. You modify your code to ask the cache for results, and, if you don’t get a result, then you go get it from the database (or whatever) as usual.

But what about situations like mine? I have one server. And I use MySQL query caching. But it turns out it’s useful. (One argument for using it is that you can just run multiple clients on a single server to render moot any problems with using more than 4GB on a 32-bit system… But I’m not lucky enough to have problems with not being able to address my memory.)

MySQL’s query cache has one really irritating “gotcha”–it doesn’t catch TEXT and BLOB records, since they’re of variable length. Remembering that this is a blog, consisting of lots and lots of text, you’ll quickly see my problem: nearly every request is a cache miss. (This is actually an oversimplification: there are lots of less obvious queries benefiting, but I digress.) (WordPress complicates things by insisting on using the exact timestamp in each query, which also renders a query cache useless.) I just use SuperCache on most pages, to generate HTML caches, which brings a tremendous speedup.

But on the main page, I’m just hitting the database directly on each load. It holds up fine given the low traffic we have, but “no one uses it” isn’t a reason to have terrible performance. I’ve wanted to do some major revising anyway, so I think a rewrite in my spare time is going to experiment with using memcached to improve performance.

Simple English

Posted on by
Reply

I had someone on a forum I frequent ask me a question. It’s in broken English and he explained that he’s not a native speaker. In my reply, I tried to be sensitive to that by speaking somewhat simply. Not in a demeaning way, but in my attempts to learn Spanish, I learned very quickly that short sentences expressing one simple idea are much simpler than elaborate sentences conveying a complex range of thoughts, such as this one.

And that reminded me of the Simple English Wikipedia project, which I think should get more attention. They treat it like another language: there’s English, Spanish, Simple English, etc. A really awesome idea in my opinion. (Although the sexual intercourse page–warning, has an illustration of the process–is still pretty sketchy… Maybe because it’s not a topic that lends itself to being explained in simple, direct terms?)

I think “Simple English” is something everyone should practice, though. As some of the pages on the simple.wikipedia.org site show, writing in a simple manner does not necessarily require coming across like a dimwit.