{"id":1181,"date":"2008-09-30T11:34:51","date_gmt":"2008-09-30T15:34:51","guid":{"rendered":"http:\/\/blogs.n1zyy.com\/n1zyy\/?p=1181"},"modified":"2008-09-30T11:34:51","modified_gmt":"2008-09-30T15:34:51","slug":"wikipedia-geek","status":"publish","type":"post","link":"https:\/\/blogs.n1zyy.com\/n1zyy\/2008\/09\/30\/wikipedia-geek\/","title":{"rendered":"Wikipedia Geek"},"content":{"rendered":"<p>I noticed Wikipedia was extraordinarily slow.<\/p>\n<p>Instead of being normal and thinking, &#8220;Huh, I can&#8217;t reach Wikipedia,&#8221; I decided to investigate. One neat thing about Wikipedia is that it&#8217;s very open: not just in the sense than anyone can edit Wikipedia or that Mediawiki is open-source, but that you can view <a href=\"http:\/\/ganglia.wikimedia.org\/pmtpa\/?gw=fwd&#038;gs=Wikimedia%40http%3A%2F%2Fganglia.wikimedia.org%2F\">their Ganglia monitoring system<\/a> and even an (off-site) <a href=\"https:\/\/wikitech.leuksman.com\/view\/Server_admin_log\">server admin log<\/a>, which in this case reveals the problem. (Reversing the order of entries so that it&#8217;s chronological):<\/p>\n<blockquote>\n# 14:00 RobH: updated redirects.conf and pushed change for orphaned domains. \n# 14:01 RobH: Site is down, go me =[\n# 14:06 RobH: Pushed out old redirects.conf and restarted apaches.\n#  14:10 RobH: Site back up, slow as squids play catchup.\n<\/blockquote>\n<p>So there you have it. A broken configuration file got released, breaking all the backend Apache webservers. It was fixed, but seems that the cache is still being rebuilt, so it might be slow for a while. In the meantime, why not read up on <a href=\"http:\/\/meta.wikimedia.org\/wiki\/Wikimedia_servers#Overall_system_architecture\">the Wikipedia server cluster<\/a>? Or some <a href=\"http:\/\/www.nedworks.org\/~mark\/reqstats\/\">graphs<\/a>? For example, the <a href=\"http:\/\/www.nedworks.org\/~mark\/reqstats\/trafficstats-daily.png\">daily bandwidth usage<\/a>: pmtpa is their Tampa, FL colocation facility (serving the US) and knams is in the Netherlands (I think). (yaseo, the yellow one on the legend with no data, is a Yahoo data center in Korea.) You can see that the US cluster is hitting 3 Gbps, while the Europe cluster is exceeding 5 Gbps (!).<\/p>\n<p>In any case, Wikipedia&#8217;s back up now. \ud83d\ude1b<\/p>","protected":false},"excerpt":{"rendered":"<p>I noticed Wikipedia was extraordinarily slow. Instead of being normal and thinking, &#8220;Huh, I can&#8217;t reach Wikipedia,&#8221; I decided to investigate. One neat thing about Wikipedia is that it&#8217;s very open: not just in the sense than anyone can edit &hellip; <a href=\"https:\/\/blogs.n1zyy.com\/n1zyy\/2008\/09\/30\/wikipedia-geek\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1181","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts\/1181","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/comments?post=1181"}],"version-history":[{"count":0,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts\/1181\/revisions"}],"wp:attachment":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/media?parent=1181"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/categories?post=1181"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/tags?post=1181"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}