[Pacemaker] Percona Replication Manager - Adding a new slave

[Mark Stunnenberg]

Continuing on the mailing list (I posted it on the site since it was 
more of a addition to your blog post):

---

That's the whole reason for me to add that post on your blog, the 
functionality you describe is not working.

I had a test setup with 3 servers, which was working nicely.
Then I removed 1 server and replaced it with a fresh server. (so the 
clone-max is already on 3)

- Removed the "old" node from the config.
- I installed a server, mysql-server, made a binary copy.
- Started the slave to test replication from master.
- Installed corosync/pacemaker/mysql-RA on new server.
- Stopped mysql.
- Started corosync.

What happened was that the mysql RA did a CHANGE MASTER TO on my new 
slave, with the data it read from the clusters' "replication_info" 
variable. But the thing is, that data is set by the current master when 
it became a master. That has been over a month ago, so the master log 
files aren't there anymore, the "master log pos" and "master log file" 
are not consistent with the snapshot I made from the master to create a 
new slave. So I ended up with a new slave and the error that it couldn't 
find mysql-bin.000002 (that's the binlog that was active when the 
current master became one)

What I did was this:
- Created a new slave (again)
- Let it replicate from master to test
- Stop slave and check it's current master log file and log pos
- Use the following command to set the correct values in the 
"replication_info" variable:

crm_attribute --type crm_config --name replication_info -s 
mysql_replication -v "192.168.5.96|mysql-bin.000022|1029552725"

Now start corosync on the new slave, slave resources are started and the 
CHANGE MASTER TO will set the correct values and replication will start.

So every time I want to add a new slave to the cluster, before you start 
corosync on it, you need to set the "replication_info" variable to 
reflect the currently added slave's position in the master log file.
That's not really a big problem, it's just something that should be 
noted ;)


If I try the method you describe, the slave wants to start replicating 
from an invalid position (because of an incorrect "replication_info".

in the code I read:

mysql_start() {

..snip..

     if ocf_is_ms; then
         # We're configured as a stateful resource. We must start as
         # slave by default. At this point we don't know if the CRM has
         # already promoted a master. So, we simply start in read only
         # mode.
         set_read_only on

         # Now, let's see whether there is a master. We might be a new
         # node that is just joining the cluster, and the CRM may have
         # promoted a master before.

         master_host=`echo $OCF_RESKEY_CRM_meta_notify_master_uname`
         if [ "$master_host" -a "$master_host" != `uname -n` ]; then
             ocf_log info "Changing MySQL configuration to replicate 
from $master_host."
             set_master
             start_slave
             if [ $? -ne 0 ]; then
                 ocf_log err "Failed to start slave"
                 return $OCF_ERR_GENERIC
             fi
         else
             ocf_log info "No MySQL master present - clearing 
replication state"
             unset_master
         fi

.. snip ..
}

The cluster cannot know what to set the "replication_info" to, since it 
doesn't know where the slave is and if it's correct, so it needs to be 
done manually.

Correct me if I'm completely wrong here! :)

Regards,
Mark Stunnenberg



---

Hi Mark,
the discussion move to the pacemaker mailing list.  I'll answer the 
slave since it is a common question.  Adding a new slave is very easy, 
if the currnet master is the same as the one in the replication_info, 
binlog file and position are ignored. So what you do is the following.

- Setup replication with the master
- Stop mysql
- Start Pacemaker on the new server and increase clone-max by 1.  This 
will start MySQL and it will be a slave of the current master.

Please ask your next question to the pacemaker mailing list: 
pacemaker at oss.clusterlabs.org

Regards,

Yves



On 12-03-07 01:11 PM, Mark Stunnenberg wrote:
 > It's been pretty quiet here for a while now :)
 >
 > I've to add something I ran into while I had some maintenance planned 
 > on the cluster. What I wanted to do is add another server to the
 > cluster.
 >
 > The "replication_info" config setting is only changed when a new
 > master is elected. So how to add a new slave?
 >
 > Create a new slave by dumping/binary copying or any other way your
 > normally prepare a slave. Now change the "replication_info" manually
 > in corosync to the new binlog and logpos position from when you took 
 > the dump/copy.
 >
 > From within corosync itself you'll get an error when you try to edit
 > it, so use the command that the mysql RA uses, which is:
 > crm_attribute --type crm_config --name replication_info -s
 > mysql_replication -v "||"
 >
 > Now start corosync on your slave and let it join the cluster, it will
 > now correctly start slaving from the position you made the snapshot.
 >

-- 
Regards,

Yves
skype: y.trudeau
aim: ytrudeaumysql
gtalk/Jabber: trudeau.y at gmail.com
email: yves at percona.com
Phone: (408) 457-0699
Flat-rate 24x7 support for MySQL<http://tinyurl.com/mysql-support>