For those of you who don’t monitor police scanners regularly, I’d like to introduce what can be considered a fairly scary fact: their computer systems go down all the time.

Where it usually comes up is when they try to run a license plate or a person, or to query NCIC or similar. The officer calls it in and waits a few minutes, before the dispatcher calls back that the (remote) system is down. When you’re monitoring multiple neighboring towns, you’ll often notice that they all lose it at once. The backend servers are going down.

This drives me nuts. It’s usually not a huge deal, but now just imagine that you’re the police officer, and the guy you pull over, but can’t run through the system, actually has a warrant out for his arrest. For murdering a police officer. But you have no clue, because the system is down. Of course this is extreme, but it’s always been said that traffic stops are actually the most dangerous and unpredictable things an officer does. They never know whether it’s a nice old lady or someone with a warrant out for their arrest. A decent amount of arrests come from pulling people over for traffic violations and finding subsequent violations, like cocaine or guns, or an outstanding warrant.

My webserver sits in Texas on what’s basically an old desktop system. And it seems to have better uptime than these systems. As biased as I am in favor of my blogs, even I will admit that police databases are more important. Further, if my blogs were routinely unreachable, I’d be furious with my hosting company. Why is it tolerated when this happens?

Databases are fairly easy to replicate. Put a “cluster” of database nodes in a datacenter. You’re protected against a hardware failure. Of course, the data center’s still a single point of failure. So put another database node in a separate datacenter. That alone is probably all you’ll ever need. But you can keep turning up more database nodes in different locations as budget permits. (I suspect budget is the limiting reactant.)

But you can take it one step further. Set up another database node, not in a lonely datacenter, but in a large dispatch facility. (The MA State Police apparently run a very large 911 answering center.) So they get a database node there, that doesn’t answer public queries, but that receives updates from other database servers. And, in the event of some sort of catastrophic failure, remote dispatchers can call up and request that something be run.

I’m just really bothered that people seem to find it acceptable that, probably at least once a week, the system is unreachable for quite some time.