[Pacemaker] MySQL startup slow on OCFS2

[daniel qian]

I followed this link to setup a two-node cluster on Ubuntu 10.4 - https://wiki.ubuntu.com/ClusterStack/LucidTesting#Pacemaker,%20drbd8%20and%20OCFS2%20or%20GFS2

Everything is working fine except for running MySQL on both nodes with MySQL datadir set to the drbd based OCFS2 disk space.  Everytime I run command 'service mysql start'  on the second node to start up MySQL it takes a much longer time than it does on the first one to start. I tried changing the order of the two nodes to start MySQL it is always the node that starts the second MySQL. 

I set apparmor for msyqld to complain mode and noticed a lot of these lines being produced in /var/log/syslog during the delay:

May 26 10:10:30 kappa kernel: [55355.685031] type=1502 audit(1274883030.944:13):  operation="mknod" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="c::" denied_mask="c::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.884676] type=1502 audit(1274883031.144:14):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="rwc::" denied_mask="rwc::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.884752] type=1502 audit(1274883031.144:15):  operation="unlink" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="d::" denied_mask="d::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.885013] type=1502 audit(1274883031.144:16):  operation="mknod" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="c::" denied_mask="c::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.885134] type=1502 audit(1274883031.144:17):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="rwc::" denied_mask="rwc::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.885180] type=1502 audit(1274883031.144:18):  operation="unlink" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="d::" denied_mask="d::" fsuid=0 ouid=0 name="/mnt/mysql/kappa.lower-test"
May 26 10:10:31 kappa kernel: [55355.890432] type=1502 audit(1274883031.148:19):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="r::" denied_mask="r::" fsuid=200 ouid=200 name="/mnt/mysql/mysql/plugin.frm"
May 26 10:10:31 kappa kernel: [55355.890643] type=1502 audit(1274883031.148:20):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="rw::" denied_mask="rw::" fsuid=200 ouid=200 name="/mnt/mysql/mysql/plugin.MYI"
May 26 10:10:31 kappa kernel: [55355.890725] type=1502 audit(1274883031.148:21):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="rw::" denied_mask="rw::" fsuid=200 ouid=200 name="/mnt/mysql/mysql/plugin.MYD"
May 26 10:10:31 kappa kernel: [55355.925736] type=1502 audit(1274883031.184:22):  operation="open" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="rw::" denied_mask="rw::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:36 kappa kernel: [55360.934856] __ratelimit: 15 callbacks suppressed
May 26 10:10:36 kappa kernel: [55360.934861] type=1502 audit(1274883036.194:28):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:37 kappa kernel: [55361.936517] type=1502 audit(1274883037.196:29):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:38 kappa kernel: [55362.938148] type=1502 audit(1274883038.197:30):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:39 kappa kernel: [55363.939795] type=1502 audit(1274883039.198:31):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:40 kappa kernel: [55364.941453] type=1502 audit(1274883040.200:32):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:41 kappa kernel: [55365.943089] type=1502 audit(1274883041.202:33):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:42 kappa kernel: [55366.944713] type=1502 audit(1274883042.204:34):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"
May 26 10:10:43 kappa kernel: [55367.946398] type=1502 audit(1274883043.205:35):  operation="file_lock" pid=8666 parent=1 profile="/usr/sbin/mysqld" requested_mask="wk::" denied_mask="wk::" fsuid=200 ouid=200 name="/mnt/mysql/ibdata1"              


Does anyone know what is going on here with mysql?


Thanks,
Daniel