Building a stratum 1 NTP server on EC2

I won’t even try to pretend I’m not a huge geek here…

I’ve run public NTP servers for ages, and been somewhat intrigued ever since a dedicated server ended up misclassified as being in Brazil and got an enormous volume of traffic, but without impacting performance. I have a DigitalOcean droplet in Singapore that’s serving a few terabytes a month of NTP queries; the pool tells me it’s about 3% of pool traffic for all of Singapore.

NTP servers are categorized into strata, essentially indicating how far down in the chain a clock is. A stratum 1 server gets its time directly from a (non-NTP) reference source, and a system syncing to a stratum 1 server becomes stratum 2, and so forth. (As an aside, strata don’t actually have enormous significance: a local stratum 3 is probably more accurate than a stratum 1 on the other side of the globe.)

For years, EC2 instances have had Time Sync available over a link-local address. This, incidentally, has proven the point about server stratum not being everything: the Time Sync server is generally at stratum 3, but I infer that everything up to stratum 1 occurs within the availability zone, if not data center, level. Given the link-local Time Sync address and some good stratum 1 clocks within the country, my servers running in AWS typically end up syncing to Time Sync. I have a couple servers in the pool at stratum 4 as a result. It’s further down “the chain,” but it’s a particularly good chain.

But more recently, Amazon has been making PTP available on certain instance types in certain regions. While PTP can be succinctly described as a more precise version of NTP (it is, after all, the Precision Time Protocol), it’s a lot more precise, making use of hardware timestamping the whole way, and can therefore achieve sub-microsecond accuracy.

And so, I spun up an instance in Malaysia and put it in the pool. I’ve kept it at only 512 kbps because there were only 3 IPv4 servers in all of Malaysia, and AWS bandwidth pricing is obscene. It has the PTP Hardware Clock (PHC) set up as a reference source, and is reporting offsets in the range of a handful of nanoseconds.

I went a step further and configured it for NTS. This required a hostname, and in trying to use Porkbun to find a novel domain name, I stumbled over the fact that ntpservers.org was open. Obviousy, I went ahead and registered it.

So now there is malaysia-1.ntpservers.org online, as a stratum 1 NTP server in Malaysia supporting NTS.

As an aside: I ended up bringing up malaysia-2.ntpservers.org as well in a (seemingly paradoxical) attempt to save money. It’s a cheap instance as a provider including 2TB of bandwidth for under $10/month (USD). Amusingly, its default config synced to the pool and had selected my stratum 1 server. I cleaned the config up slightly to point to some stratum 1 clocks in Japan an Singapore, and it’s been fielding a lot more traffic from the pool at a higher bandwidth setting.

I have not yet listed it anywhere outside registering it in the pool, because it’s currently more of a novelty than a long-term committment. I am tempted to eventually withdraw malaysia-1 from the pool but list it as an available stratum 1 supporting NTS on the relevant sites to hopefully cut down bandwidth costs and make it reasonable to maintain long-term.

Right now the other AWS regions supporting PTP instances are all areas well-served by stratum 1 clocks: Tokyo and the United States. It will be interesting to see if it becomes available in, say, India, where the available options are more limited.

It’s back!

While the blogs aren’t updated very often, at some point they stopped working an just displayed a 502 error.

If I’m being honest, I have not idea what happened. The old setup was incredibly convoluted: a CDN in front of an Amazon ALB, in front of an oversized EC2 instance, using EFS for storage and an RDS instance for the database. The site was extremely behind on updates, and relied on a couple of separate custom plugins to properly render everything. At some point something in the chain stopped working.

Rather than debug it, I did something I’ve wanted to do for ages: I ripped everything apart and just moved it to a cheap shared hosting account at Dathorn. (Where “cheap” refers to price, not quality. I love Dathorn.) I kept Bunny in front as a CDN, in large part because the blog is basically static these days and because my prepaid $25 account will still last more than a year at current traffic levels. I even fixed (I think?) an issue where the fonts on my blog were being pulled from a defunct @font-face provider. Don’t expect a flurry of updates here, but with any luck, the site will stay online now!

The myth of Prunella Shriek

This week we are in Ocean City, Maryland, and picked up some donuts from a popular local place, The Fractured Prune. Their donut box is extraordinarily bizarre:

We had a discussion (including abundant use of “WTF”) about this, including why you would name a donut place after a prune, what it even means to “fracture” a prune, and why their mascot needed to be so creepy. (Also, the curious misuse of quotes around “EM”)

The Washington City Paper answered most of our questions in a hilarious post, explaining the explanation given by Tom Parshall, owner:

Back in the late 1800’s a woman named Prunella Shriek had owned much of the land in that area. Tom didn’t think too much about it until one day he was browsing through the Baltimore Library and came across a book titled Ocean City Love. As he looked through it, he was thrilled to see Prunella Shriek’s name with a paragraph devoted to her.

Even though she was in her seventies, she was the only woman who competed with men in traditionally men’s sports such as ice skating races & skiing competitions. She was an excellent tennis player, and was county ping-pong champion circa 1895. Being older and somewhat brittle, she often returned from some of the more vigorous competitions in a wheel chair or on crutches. Thus the townspeople began lovingly to refer to her as “Fractured Prunella.” When Tom read this, a voice in his head said, “Hello”, he would name his enterprise after this wonderful & spunky old lady, and call it “The Fractured Prune®.”

The author, Tim Carman, is seemingly the only person on the Internet to have not accepted this as truth on its face, and he did what any good journalist ought to:

I decided to call up the current owner, Sandy Tylor, and find out whether the story was fact or just marketing BS. “That’s what was told to me when I purchased it,” says Tylor, who bought the small-but-delicious doughnut chain in 1994 with her daughter and son-in-law. “That’s what I tell people.”

Denver’s 5280 cites their marketing story uncritically, as does Penn Live. Las Vegas Eater does it too, and of course Fox didn’t fact check it. Nor does the Baltimore Sun. I’ve got about a dozen more tabs reporting the same story and have lost interest in citing any further examples.

Why so many tabs? Because I was trying to find any evidence that a person named Prunella Shriek existed.

Ancestry.com has no matches:

(The date of birth is a crude guess based on the fact that she was described as being in her 70s in the late 1800s.) Contemporary white-page sort of searches don’t support there being anyone with the surname “Shriek” in Ocean City.

What about Ocean City Love, the book Tom Parshall cites for where he learned about Prunella and her penchant for sporting injuries? Baltimore Public Library doesn’t seem to have any such title in its collections, and I can’t find any evidence of its existence in the Library of Congress.

So I agree with Gretchen Kurtz’s review in Westword, that their donuts are good, even if the name “sounds more like a joke than anything else.” I’m fairly confident that Prunella Shriek never existed.

Skyking, Skyking, Do Not Answer

After some aimless browsing, I somehow came to be reading about the Jim Creek VLF facility in Washington State, used for communications with (or, more accurately, communications to) submarines. Even with miles and miles of wire antennas in the air, and even with more than a Megawatt out, the signal isn’t able to penetrate very far in saltwater.

This reminds me of the broadcasts carrying instructions to nuclear-equipped Air Force resources. There are both Emergency Action Messages and Skyking broadcasts, with the latter apparently having higher priority. They’re and are, of course, encrypted messages.

They’re arguably similar to the various numbers stations, which broadcast information encoded with a one-time pad and are used to communicate information to people in the field. For example, see The Lincolnshire Poacher, which begins with the titular melody being played repeated, presumed to be an attention signal to aid in tuning. Many of the numbers stations use a recorded voice reading off numbers or letters. Some of what we know was confirmed when the US tried five Cuban spies.

Murray G2250030 Snowblower Parts

A neighbor gave me an old snowblower she didn’t need, a green 22″ Murray 5HP model. A label on the back identifies it as model number G2250030. I’ve been trying to fix it up and get it running on the side, but it’s been a bit of a challenge. I cannot for the life of me find a manual.

https://partsandservice.com/html/Murray/st/stg2250-030.html at least has some parts diagrams. From there and some Amazon exploration, I’ve found some replacement parts. Note that I’ve just now ordered these; I’ll come back and update this to reflect whether the parts are correct or not.

Shear Pins

As best as I can tell, these are identified as part 9524, “Screw, 1/4-20×1.75” and part 3943, “spacer, sleeve” in the auger housing assembly parts diagram.

I think these are what you want.

This eBay listing suggests that parts 1501216 (and 1501216MA), 9524MA, and 3943MA are equivalent, and that Craftsman used the 9524 part number as well.

Whatever you buy, make sure you get the spacer. Some reviews on other parts (not linked here) indicate that people cannot find the spacer individually.

Belts

I plan to replace both drive belts while I have the thing apart.

The drive belt (turns the wheels) is part 579932, described as “Belt V 3L 33.13LG”. I have this in my Amazon cart right now. (Note that the “MA” prefix is often added to Murray part numbers, so it’s also known as 579932MA.)

The auger drive belt is 581264, described as “Belt, V 4L 35.23LG”. I have this on order.

Auger throttle cable

The cable to engage the augers is the faulty part I’m actually trying to replace. It’s part 340373, “Cable, auger.” It appears that this part number has been replaced by 761400MA, which is also apparently equivalent to 760150, 761153. See this part, which is running about $38.

What is repmgr on a Mac?

I now have an IT-managed Mac at work, which means that I had to cede a little control and allow weird stuff I don’t understand to run on it.

I was disturbed today to find a root-owned process, repmgr, using a bunch of CPU. Some quick Googling suggests that repmgr is for Postgres replication, which is problematic because I don’t use Postgres and don’t have it installed.

It turns out that there’s another, unrelated repmgr, though! /Applications/Confer.app/Contents/MacOS/repmgr is the path to the running executable for me. repmgr is also the name of a component of Confer / Carbon Black, some sort of enterprise security software for Macs. I haven’t had the chance to figure out exactly what it does, but based on some of the strings in it, I suspect it’s short for “Reputation Manager.”

WR Communications Manuals

Someone on the repeater-builder mailing list mentioned that they were looking to get rid of a bunch of manuals for old radio equipment by WR Communications, Ltd., a defunct two-way radio company from Canada. (See their Industry Canada listing here.)

For some reason, despite having not previously heard of the company, I couldn’t bear to see a bunch of documentation get thrown away, so I arranged to have them shipped to me. I’m planning on scanning all of them in and OCR’ing them. Even with a fast scanner with a document feeder, it’s going to take a bit.

I’m currently in possession of the following manuals. The last number is seemingly a document number they used for some purpose.

  • 9T90B10 UHF Transmitter (72)
  • RTL-54 Trunk Adapter (21)
  • 3R90B VHF Rx Instruction Manual (60)
  • 9R90B UHF Receiver (71)
  • RTA-54 CTCSS Encoder/Decoder (18)
  • 3R93B VHF Receiver (77)
  • WR Type Number Designation Scheme (Rackmount 90 Series) (79)
  • 3T93A6 UHF Transmitter (75)
  • 9R93A UHF Receiver (76)
  • RCP-54 DC Power Control, Sub-Assembly No. 14.1284 (80)
  • DTMF Decoder Latch Board (81)
  • M-90 Test Set Schematic Diagram (12)
  • Photos (39)
  • RIA-54 Intercom Adapter (16)
  • RLA-54A DC Line Adapter (17)
  • 9T90B4 UHF Tx Instruction Manual (61)
  • 9T90B4 UHF Tx Instruction Manual (58)
  • 3T90B4 VHF Tx Instruction Manual (59)
  • WR-194-70W Power Amplifier Section Schematic (27)
  • Repeater Control RCL-54B Sub-Assembly No. 14.0571 (?)
  • 3T90B-15 VHF Transmitter (70)
  • Rack Mount Station Power Supply Schematic Diagram (9)
  • 95 Rack Mount PA Power Supply Schematic Diagram (13)
  • TR-100 VHF Antenna Switch Schematic Diagram (22)
  • TT-4 and TT-4A Time Out Timer Lockout (23)
  • WR-194 Receiver Module (24)
  • WR-194 Transmitter Module (25)
  • WR-194-4W Transmitter Module (26)
  • 9T95A50 Power Amplifier Section Schematic Diagram (55)
  • Rack Mount Station TX Module WR-454 (44)
  • A folder with miscellaneous DTMF latch decoder schematics and info
  • Unlabeled folder for 9R90A-0 Receiver Module
  • WR-494 Transmitter Module (36)
  • A bound manual containing:
    • 9T90B4 UHF Transmitter Instruction Manual
    • 9R90C UHF Receiver Assembly No. I2.0283 (12.0283 ?)
    • Repeater Control RCL-54B Sub-Assembly No. 14.0571
    • CTCSS Encoder-Decoder RTA-54B Sub-Assembly No. 14.0581
    • CTCSS Tone Control RTC-54B Sub-Assembly No. 14.0601

Low-Overhead Linux Networking

I want to do more posts here that are quick roundups of interesting things I’ve read, rather than rambling commentary.

  • DPDK, the Data Plane Development Kit, permits a userspace application in Linux or BSD to bypass the kernel and directly interact with a network card. Handling interrupts to pass data from a 100 Gbps network card through the kernel apparently performs poorly.
  • Selectel’s “Introduction to DPDK: Architecture and Principles” looks like a detailed walkthrough of how it works. (I only skimmed it on the train tonight though.)
  • FD.io is a Linux Foundation project implementing a software dataplane employing vector packet processing (VPP). Its What is VPP? page is a good intro.
  • networking-vpp brings support to OpenStack networking.
  • Ligato and Contiv-VPP bring VPP to Kubernetes (et al.)
  • I’ve run up against mentions of YANG, which is “Yet Another Next Generation” DSL for modeling network data.
  • Intro to networking-vector packet processing is based on a 2018 FOSDEM talk. The Youtube video really helped things “click” for me.

2018 Camaro SS review

Last week, I had to have my car towed into the shop—and then from there to the dealership. (It felt sort of like when someone is taken by ambulance to a local hospital and then taken by medical helicopter to a larger hospital.) The dealership didn’t have any loaners, so they called Enterprise.

Now, Enterprise isn’t a bad company, and their cars aren’t necessarily undesirable, but no one bringing their car to a BMW dealership is hoping for an Enterprise rental. Like any other rental company, the car is always a little tin can that somehow doesn’t have GPS in 2018.

So imagine my surprise when they gave me the keys to this:

I didn’t know much about Camaros, but it turns out that, even more improbably for a rental car, this was one of the good ones, the Camaro 1SS with a V8 making 455 horsepower. Sitting in the driver’s seat with the Enterprise employee in the passenger’s seat as we completed paperwork, it was hard to contain my grin as I pushed the start button and the car let loose a throaty growl.

Once I filled out all the paperwork promising to treat the car gently and the Enterprise employee took off, I tried to at least leave the lot before doing all the things I said I wouldn’t. In my car, you need to give the pedal a fair bit of pressure when the light turns green. I almost think of it as slack in the pedal. Of course you don’t slam the pedal to the floor, but if you’re too ginger, people are going to start honking. Getting a heavy car rolling takes a bit of gas.

At first blush, the Camaro seemed the same. Give it a tiny little bit of gas, and the engine doesn’t deliver much power. So you do what I do in my car—give it a fair bit of pressure. And—oh shit!—that’s how I peeled out of the lot. With way more power than I expected, I let off on the gas in a panic, leaving me rolling through the intersection slowly and looking even more like I shouldn’t be trusted with this car. It didn’t take long to find a better balance, but it certainly wasn’t a great start.

Driving is a blast, though. It’s loud, and packs an awful lot of power. Whether it’s driving around town or passing a slowpoke on the highway, it doesn’t take much to get this thing going, and it’s hard to not have fun while doing it.

An interesting thing I discovered is that the car seems to shut down half of the cylinders when you’re cruising along at a steady speed, which is probably why the thing gets more than 20mpg:

(I don’t know who took this photo, but I’m sure they were on a dyno.)

But go to pass someone, and those extra cylinders come back in the blink of an eye. You’d never know they weren’t running if not for that screen showing it.

The specs show a 0-60 time of about 4 seconds. I didn’t dare try to run a timer while pinned to the back of my seat on the onramp, but I don’t doubt that number. I never got to the end of an onramp wanting for speed, but did often have to let off the gas long before the end because I had surpassed 80.

The ride wasn’t _un_comfortable, but you definitely felt every little bump in the road… And New England roads aren’t exactly smooth in early spring.

Interior

I had no trouble with headroom, but the car does sit quite low which probably helps. Audio controls on the steering wheel help you keep your eyes on the road, though I missed the volume controls my car had. The steering wheel felt like leather, imitation or not. The bottom of the steering wheel is squared off for reasons I don’t entirely follow; perhaps to help with legroom getting in. It was a little odd turning into sharp spaces where the steering wheel wasn’t round, but I tend towards so it didn’t come up too often.

A decently-sized LCD in the center console shows everything, from music to climate control settings. Climate control has dedicated hardware buttons, and adjusting temperature or fan speed is done by turning those huge center vents. An overlay on the LCD simply shows where they’re set. The screen has a slight downtilt which seems to prevent the glare that plagues my car’s center screen.

In the instrument panel, the tachometer and speedometer are both large analog meters. Their layout is the opposite of a BMW, but it’s not like it’s hard to figure out which is which. The gauges in the middle are digital displays styled to look like analog meters. A compass readout is front and center, and was just a tad distracting while turning.

The rearview mirror was adequate but remarkably small. At the same time, that also describes the rear window, so perhaps they’re paired well. Flipping the mirror to night mode is done by hand (soooo 2015!), and the button is recessed directly behind the OnStar button. Pressing the OnStar button by mistake makes a loud chime that will freak you out. (Ask me how I know.)

Visibility overall is poor. The front windshield is remarkably short, but gives an adequate view of the road ahead. Side visibility past 90 degrees, though, is atrocious. Trying to take a sharp left while watching for traffic from the right required crazy contortions with my head against the windshield to see past the passenger’s seat. You need to have your mirrors adjusted correctly in this, because peripheral vision is poor.

One review called the rear seats “vestigial,” which is the perfect word to describe them. I used them as a shelf to hold my umbrella. “Legroom” in the rear is comical; reaching to get my umbrella when it slid onto the floor, I didn’t have enough arm room to pick it up. Perhaps if the driver and front passenger have very short arms and legs and pull their seats way forward, you could seat four if the rear passengers were emaciated. I really can’t imagine when you could fit anyone in the back, unless they were maybe in a car seat. (And if you’ve got a toddler, I humbly submit that perhaps a Camaro isn’t the right car for you at this stage of your life.)

In short: having the Camaro for a weekend was a blast. And I certainly wouldn’t mind having one as an occasional fun thing to drive. But by the time my car had two functioning axles again on Monday, I was happy to get something more practical back. Visibility and comfort are surprisingly important.

Bulk-editing HTML files with images containing a # sign

A friend of mine recently pinged me with a problem he’s having. I wanted to share the problem and its solution, both because I hope it will help him and maybe others, and because my initial attempts at the problem were way more complicated than they needed to be.

The problem

My friend used some software to generate a whole bunch of HTML reports, and the reports contained images with names like “Project #1 – Blah/Image #1.jpg” or such. None of the images showed up.

He was savvy enough to recognize that the # (octothorp, pound sign, number sign, “hashtag symbol,” etc.) was the problem, but this was little comfort: with about 100 generated HTML files, each with six images, editing it all was perhaps an entire day’s work.

The problem is that the pound-sign is a special character in URLs, the fragment identifier. (I admit, I had to look up what it’s actually called.)

The over-engineered fix

What I struggled with was this: I could easily write a quick shell script to rename all the folders, and then mass-edit the HTML files. It would probably be 5-10 minutes of work.

What I couldn’t do easily, though, was write a Ruby (or whatnot) script without seeing the files and then get my friend to run it, on what I presume is a Windows computer without a Ruby interpreter. It’s one of those problems that I could very easily fix for myself, but communicating it or getting it to run for someone else is tremendously more complicated.

So I proposed that he just create a Zip file with the entire folder contents, including 600 images, and send it to me to fix.

This is a bad fix for two reasons. One, emailing 600 images is crazy. But two, there’s a much easier fix that hadn’t occurred to me.

Tech-savvy, non-programmer fix

Having suggested the above fix at first, I mulled over something else he said: this was one of those things he wasn’t sure how to even search for on Google, because he didn’t really know the terms. I think this is a problem we’ve all faced at one time. (For example, when I dropped a tiny little weird-shaped screw into my carpet and wanted to buy a replacement. If you have no idea what the thing is called, and I didn’t, finding it online will be even harder than finding it in your carpet.)

I wondered: what could you search for? I tried “how to bulk-edit HTML files” or something of the sort. It likely wouldn’t have gotten him the details he needed. But it helped me realize an easier fix!

There’s an easy way to bulk-edit text files, beyond writing a script to do it. And it’s something that programmers in particular should be familiar with: text editors.

And then something else occurred to me: you don’t actually need to rename the files! You just need to percent-encode the # in the URL!

So this suddenly becomes a much easier solution to relay.

  • Back up your work!
  • Download a free and reputable text editor. Notepad++ is highly-regarded on Windows, but I’m not a Windows user. I use Sublime Text on my Mac, and I still have a TextMate install as well.
  • Open the entire folder in the text editor.
  • Use the editor’s global search-and-replace / find-and-replace functionality. In Notepad++, it looks like it’s called Find in Files.
  • Make sure you really do have the project backed up; global search and replace on a project can be a real pain to undo if it goes wrong.
  • Search for # in all HTML files (*.html) and replace it with the percent-encoded representation, %23. This can look ugly: “Photos #1/pic.png” becomes “Photos %231/pic.png”. (The space should technically be %20, because a space is also a special character, but browsers are better about figuring that out.) Make sure “Replace all” or the equivalent is selected.
  • Ideally, test it in one file, save that file, and verify that the images now show up.
  • Let it roar through all of the files for you.

There’s one other risk worth noting, which is that the # sign could be used elsewhere in the file, particularly in its correct use as a fragment identifier, like if there was a link to different sections within the page. In that case, you would need to be slightly more selective in your search and replace, and this is where it gets gory. You’d need to find a search string that matches only the image URLs, make like “Photos #”, and then replace it with the same thing, but with the “#” changed to a “%23” — i.e., search for “Photos #” and replace it with “Photos %23”.