[Pacemaker] MySQL startup slow on OCFS2

[daniel qian]

On 2010-05-26, at 2:26 PM, Lars Marowsky-Bree wrote:

> On 2010-05-26T10:26:35, daniel qian <daniel at bestningning.com> wrote:
> 
>> I followed this link to setup a two-node cluster on Ubuntu 10.4 - https://wiki.ubuntu.com/ClusterStack/LucidTesting#Pacemaker,%20drbd8%20and%20OCFS2%20or%20GFS2
>> 
>> Everything is working fine except for running MySQL on both nodes with MySQL datadir set to the drbd based OCFS2 disk space.  Everytime I run command 'service mysql start'  on the second node to start up MySQL it takes a much longer time than it does on the first one to start. I tried changing the order of the two nodes to start MySQL it is always the node that starts the second MySQL. 
>> 
>> I set apparmor for msyqld to complain mode and noticed a lot of these lines being produced in /var/log/syslog during the delay:
> 
> Does it go faster if you disable apparmor for the test?

If apparmor is on for mysqld, mysql wont even start. I have to either stop apparmor or set it in complain mode for mysqld.

> 
>> Does anyone know what is going on here with mysql?
> 
> How much slower is "slower"? OCFS2 obviously introduces some overhead if
> you have lock contention (ie, trying to modify the same file system
> objects, such as a directory).
> 
> Which versions? How are you controlling ocfs2?
> 
> 


I have not started any IO test on the file system yet. The slowness I observed is only when I start mysql and it takes about one minute to start a empty mysql install.

root at kappa:~# dpkg -l | grep -i ocfs
ii  ocfs2-tools                         1.4.3-1ubuntu0ppa4                tools for managing OCFS2 cluster filesystems       

root at kappa:~# crm configure show
node dns
node kappa
primitive resDLM ocf:pacemaker:controld \
        op monitor interval="120s"
primitive resDRBD ocf:linbit:drbd \
        params drbd_resource="disk0" \
        operations $id="resDRBD-operations" \
        op monitor interval="20" role="Master" timeout="20" \
        op monitor interval="30" role="Slave" timeout="20"
primitive resFS ocf:heartbeat:Filesystem \
        params device="/dev/drbd/by-res/disk0" directory="/mnt" fstype="ocfs2" \
        op monitor interval="120s"
primitive resO2CB ocf:pacemaker:o2cb \
        op monitor interval="120s"
ms msDRBD resDRBD \
        meta resource-stickines="100" notify="true" master-max="2" interleave="true"
clone cloneDLM resDLM \
        meta globally-unique="false" interleave="true"
clone cloneFS resFS \
        meta interleave="true" ordered="true"
clone cloneO2CB resO2CB \
        meta globally-unique="false" interleave="true"
colocation colDLMDRBD inf: cloneDLM msDRBD:Master
colocation colFSO2CB inf: cloneFS cloneO2CB
colocation colO2CBDLM inf: cloneO2CB cloneDLM
order ordDLMO2CB 0: cloneDLM cloneO2CB
order ordDRBDDLM 0: msDRBD:promote cloneDLM
order ordO2CBFS 0: cloneO2CB cloneFS                                     



Thanks,
Daniel