I noticed Wikipedia was extraordinarily slow.

Instead of being normal and thinking, “Huh, I can’t reach Wikipedia,” I decided to investigate. One neat thing about Wikipedia is that it’s very open: not just in the sense than anyone can edit Wikipedia or that Mediawiki is open-source, but that you can view their Ganglia monitoring system and even an (off-site) server admin log, which in this case reveals the problem. (Reversing the order of entries so that it’s chronological):

# 14:00 RobH: updated redirects.conf and pushed change for orphaned domains.
# 14:01 RobH: Site is down, go me =[
# 14:06 RobH: Pushed out old redirects.conf and restarted apaches.
# 14:10 RobH: Site back up, slow as squids play catchup.

So there you have it. A broken configuration file got released, breaking all the backend Apache webservers. It was fixed, but seems that the cache is still being rebuilt, so it might be slow for a while. In the meantime, why not read up on the Wikipedia server cluster? Or some graphs? For example, the daily bandwidth usage: pmtpa is their Tampa, FL colocation facility (serving the US) and knams is in the Netherlands (I think). (yaseo, the yellow one on the legend with no data, is a Yahoo data center in Korea.) You can see that the US cluster is hitting 3 Gbps, while the Europe cluster is exceeding 5 Gbps (!).

In any case, Wikipedia’s back up now. 😛