{"id":1542,"date":"2009-01-27T20:29:08","date_gmt":"2009-01-28T00:29:08","guid":{"rendered":"http:\/\/blogs.n1zyy.com\/n1zyy\/?p=1542"},"modified":"2009-01-27T20:29:08","modified_gmt":"2009-01-28T00:29:08","slug":"mysql-replication-lessons-learned","status":"publish","type":"post","link":"https:\/\/blogs.n1zyy.com\/n1zyy\/2009\/01\/27\/mysql-replication-lessons-learned\/","title":{"rendered":"MySQL Replication Lessons Learned"},"content":{"rendered":"<p>A couple things I ran into today that I want to keep searchable here in case I run into them again, and that I figured might be useful to someone else someday:<\/p>\n<p>Let&#8217;s say that you take down a MySQL server that&#8217;s a replicated slave to do a memory upgrade, and it takes a really long time to shut down, and then you find that the machine doesn&#8217;t like the &#8220;new&#8221; DIMMs, so you throw the old ones back in and power it up. Just hypothetically. You then restart MySQL and issue the <tt>START SLAVE<\/tt> command, but it dies with an error:<\/p>\n<blockquote><pre>\n090127 14:53:17 [ERROR] Failed to open the relay log '.\/mysqld-relay-bin.000023' (relay_log_pos 23726)\n090127 14:53:17 [ERROR] Could not find target log during relay log initialization\n<\/pre><\/blockquote>\n<p>The relay log and position were both wholly wrong. I poked around, and found a lot of people who ran into this; it seems to be a data corruption issue, but also happens occasionally on a reboot. There&#8217;s a bunch of suggested fixes out there that don&#8217;t actually work. One thing that <i>does<\/i> work, though, is <a href=\"http:\/\/hoschie.org\/b2evolution\/index.php\/2008\/02\/15\/mysql_failed_to_open_the_relay_log\">deleting the relay logs<\/a> on the slave. (Any time someone on the Internet tells you to delete a file, you should, of course, think &#8220;move to another directory&#8221; instead so you can undo it if need be.) Once I deleted the relay logs, it started right up.<\/p>\n<p>Lesson #2? Now you&#8217;re about 6,000 seconds behind the master, and the replication lag counter is going down at a rate of about 1 second per second. You can wait a couple hours. That seems pretty pathetic, though.<\/p>\n<p>If your to-do list reads, &#8220;1.) Get slave running, and then 2.) Fine-tune my.cnf, currently stolen from another machine,&#8221; there&#8217;s a chance that you have <tt>sync_binlogs=1<\/tt> set. This is bad for two reasons: the first is simply because of what it&#8217;s designed to do: flush the binlog to disk on every write. This is very safe, but also very slow. But the second reason it&#8217;s bad is that it&#8217;s apparently <a href=\"http:\/\/www.mysqlperformanceblog.com\/2009\/01\/21\/beware-ext3-and-sync-binlog-do-not-play-well-together\/\">especially bad<\/a> on ext3, so it&#8217;s doubly important to not use this option, at least not when write performance is important.<\/p>","protected":false},"excerpt":{"rendered":"<p>A couple things I ran into today that I want to keep searchable here in case I run into them again, and that I figured might be useful to someone else someday: Let&#8217;s say that you take down a MySQL &hellip; <a href=\"https:\/\/blogs.n1zyy.com\/n1zyy\/2009\/01\/27\/mysql-replication-lessons-learned\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1542","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts\/1542","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/comments?post=1542"}],"version-history":[{"count":0,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/posts\/1542\/revisions"}],"wp:attachment":[{"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/media?parent=1542"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/categories?post=1542"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.n1zyy.com\/n1zyy\/wp-json\/wp\/v2\/tags?post=1542"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}