[Pacemaker] Need help with Pacemaker clvm problem
Sander van Vugt
mail at sandervanvugt.nl
Thu Dec 10 12:46:38 UTC 2009
Hi,
I'm trying to set up a configuration using clvmd and it doesn't work. On
one node it comes up, on the other nodes in my three node cluster it
doesn't. These failing two nodes are BTW marked as unclean and I can't see why
that happens nor what exactly that means. I would appreciate some help in
finding the cause of this problem.
Below in and attached to this message you can find the following:
* output of crm_mon -i 1 on node 1 (one of the failing nodes)
* /var/log/mesages from the moment of the last reboot of the nd1 server
* relevant part of cibadmin -Q
Thanks in advance!
Sander van Vugt
============
Last updated: Thu Dec 10 13:35:28 2009
Current DC: nd3 - partition with quorum
Version: 1.0.3-0080ec086ae9c20ad5c4c3562000c0ad68374f0a
3 Nodes configured, 3 expected votes
3 Resources configured.
============
Node nd1: UNCLEAN (online)
Node nd3: UNCLEAN (online)
Online: [ nd2 ]
Clone Set: dlm-clone
Started: [ nd2 ]
Stopped: [ dlm:0 dlm:2 ]
Clone Set: clvm-clone
Started: [ nd2 ]
Stopped: [ clvm:0 clvm:2 ]
Failed actions:
dlm:2_monitor_120000 (node=nd3, call=6, rc=7, status=complete): not
running
o2cb:2_start_0 (node=nd3, call=7, rc=5, status=complete): not installed
-------------- next part --------------
Dec 10 13:16:47 nd1 syslog-ng[3581]: syslog-ng starting up; version='2.0.9'
Dec 10 13:16:48 nd1 ifup: lo
Dec 10 13:16:48 nd1 ifup: lo
Dec 10 13:16:48 nd1 ifup: IP address: 127.0.0.1/8
Dec 10 13:16:48 nd1 ifup:
Dec 10 13:16:48 nd1 ifup:
Dec 10 13:16:48 nd1 ifup: IP address: 127.0.0.2/8
Dec 10 13:16:48 nd1 ifup:
Dec 10 13:16:49 nd1 ifup: eth0 device: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Dec 10 13:16:49 nd1 ifup: eth0 Startmode is 'off'
Dec 10 13:16:49 nd1 ifup: eth1 device: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Dec 10 13:16:49 nd1 ifup: eth1 Startmode is 'off'
Dec 10 13:16:49 nd1 ifup: eth2 device: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Dec 10 13:16:49 nd1 ifup: No configuration found for eth2
Dec 10 13:16:49 nd1 rchal: CPU frequency scaling is not supported by your processor.
Dec 10 13:16:49 nd1 rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Dec 10 13:16:49 nd1 rchal: Cannot load cpufreq governors - No cpufreq driver available
Dec 10 13:16:49 nd1 ifup: eth3 device: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Dec 10 13:16:49 nd1 ifup: No configuration found for eth3
Dec 10 13:16:49 nd1 ifup: eth4 device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Dec 10 13:16:49 nd1 ifup: eth4
Dec 10 13:16:49 nd1 ifup: IP address: 192.168.131.1/24
Dec 10 13:16:49 nd1 ifup:
Dec 10 13:16:50 nd1 SuSEfirewall2: SuSEfirewall2 not active
Dec 10 13:16:50 nd1 ifup: eth5 device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Dec 10 13:16:50 nd1 ifup: eth5
Dec 10 13:16:50 nd1 ifup: IP address: 192.168.130.1/24
Dec 10 13:16:50 nd1 ifup:
Dec 10 13:16:50 nd1 SuSEfirewall2: SuSEfirewall2 not active
Dec 10 13:16:50 nd1 ifup: bond0
Dec 10 13:16:51 nd1 ifup-bonding: bond0 enslaved interface: eth0
Dec 10 13:16:51 nd1 ifup-bonding: bond0 enslaved interface: eth1
Dec 10 13:16:52 nd1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.444:2): operation="profile_load" name="/bin/ping" name2="default" pid=3342
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.476:3): operation="profile_load" name="/sbin/klogd" name2="default" pid=3382
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.516:4): operation="profile_load" name="/sbin/syslog-ng" name2="default" pid=3392
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.556:5): operation="profile_load" name="/sbin/syslogd" name2="default" pid=3400
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.603:6): operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default" pid=3413
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.640:7): operation="profile_load" name="/usr/sbin/identd" name2="default" pid=3421
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.679:8): operation="profile_load" name="/usr/sbin/mdnsd" name2="default" pid=3431
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.724:9): operation="profile_load" name="/usr/sbin/nscd" name2="default" pid=3432
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.784:10): operation="profile_load" name="/usr/sbin/ntpd" name2="default" pid=3433
Dec 10 13:16:52 nd1 kernel: type=1505 audit(1260447406.823:11): operation="profile_load" name="/usr/sbin/traceroute" name2="default" pid=3434
Dec 10 13:16:52 nd1 kernel: IA-32 Microcode Update Driver: v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Dec 10 13:16:52 nd1 kernel: firmware: requesting intel-ucode/06-17-0a
Dec 10 13:16:52 nd1 kernel: bnx2: eth4: using MSI
Dec 10 13:16:52 nd1 kernel: bnx2: eth5: using MSI
Dec 10 13:16:52 nd1 kernel: Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
Dec 10 13:16:52 nd1 kernel: bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.
Dec 10 13:16:52 nd1 kernel: bonding: bond0: setting mode to active-backup (1).
Dec 10 13:16:52 nd1 kernel: bonding: bond0: Adding slave eth0.
Dec 10 13:16:52 nd1 kernel: igb 0000:0a:00.0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 10 13:16:52 nd1 kernel: bonding: bond0: making interface eth0 the new active one.
Dec 10 13:16:52 nd1 kernel: bonding: bond0: first active interface up!
Dec 10 13:16:52 nd1 kernel: bonding: bond0: enslaving eth0 as an active interface with an up link.
Dec 10 13:16:52 nd1 kernel: bonding: bond0: Adding slave eth1.
Dec 10 13:16:52 nd1 kernel: igb 0000:0a:00.1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 10 13:16:52 nd1 kernel: bonding: bond0: enslaving eth1 as a backup interface with an up link.
Dec 10 13:16:52 nd1 SuSEfirewall2: SuSEfirewall2 not active
Dec 10 13:16:53 nd1 ifup: br0
Dec 10 13:16:53 nd1 ifup-bridge: br0 forwarddelay (see man ifcfg-bridge)
Dec 10 13:16:53 nd1 ifup-bridge: ... ready
Dec 10 13:16:53 nd1 ifup: br0
Dec 10 13:16:53 nd1 ifup: IP address: 192.168.1.101/24
Dec 10 13:16:53 nd1 ifup:
Dec 10 13:16:53 nd1 SuSEfirewall2: SuSEfirewall2 not active
Dec 10 13:16:53 nd1 iscsid: iSCSI logger with pid=5919 started!
Dec 10 13:16:54 nd1 iscsid: transport class version 2.0-870. iscsid version 2.0-870
Dec 10 13:16:54 nd1 iscsid: iSCSI daemon with pid=5920 started!
Dec 10 13:16:54 nd1 iscsid: connection2:0 is operational now
Dec 10 13:16:54 nd1 iscsid: connection1:0 is operational now
Dec 10 13:16:54 nd1 iscsid: connection3:0 is operational now
Dec 10 13:16:54 nd1 iscsid: connection4:0 is operational now
Dec 10 13:16:54 nd1 multipathd: 3600221900092530000000e1e4afb7067: load table [0 576716800 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 1 1 8:32 100]
Dec 10 13:16:54 nd1 multipathd: 3600221900092530000000e1e4afb7067: event checker started
Dec 10 13:16:55 nd1 multipathd: 3600221900092530000000e1e4afb7067: load table [0 576716800 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 2 1 8:32 300 8
Dec 10 13:16:55 nd1 multipathd: sdf path added to devmap 3600221900092530000000e1e4afb7067
Dec 10 13:16:55 nd1 multipathd: 3600221900092530000000e1e4afb7067: load table [0 576716800 multipath 1 queue_if_no_path 1 rdac 2 2 round-robin 0 1 1 8:48 400 r
Dec 10 13:16:55 nd1 multipathd: sdd path added to devmap 3600221900092530000000e1e4afb7067
Dec 10 13:16:56 nd1 multipathd: 3600221900092530000000e1e4afb7067: load table [0 576716800 multipath 1 queue_if_no_path 1 rdac 2 1 round-robin 0 2 1 8:48 400 8
Dec 10 13:16:56 nd1 multipathd: sde path added to devmap 3600221900092530000000e1e4afb7067
Dec 10 13:16:56 nd1 multipathd: dm-5: add map (uevent)
Dec 10 13:16:56 nd1 multipathd: dm-5: devmap already registered
Dec 10 13:16:56 nd1 multipathd: dm-5: add map (uevent)
Dec 10 13:16:56 nd1 multipathd: dm-5: devmap already registered
Dec 10 13:16:56 nd1 multipathd: dm-5: add map (uevent)
Dec 10 13:16:57 nd1 multipathd: dm-5: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: 3600221900091530d0000da294afb6f6b: load table [0 473956352 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 1 1 8:96 600]
Dec 10 13:16:58 nd1 multipathd: 3600221900091530d0000da294afb6f6b: event checker started
Dec 10 13:16:58 nd1 multipathd: sdg path added to devmap 3600221900091530d0000da294afb6f6b
Dec 10 13:16:58 nd1 multipathd: 3600221900091530d0000da294afb6f6b: load table [0 473956352 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 2 1 8:96 600 8
Dec 10 13:16:58 nd1 multipathd: dm-6: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-6: devmap already registered
Dec 10 13:16:58 nd1 multipathd: dm-9: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-8: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: 3600221900091530d0000da294afb6f6b: load table [0 473956352 multipath 1 queue_if_no_path 1 rdac 2 1 round-robin 0 2 1 8:96 600 8
Dec 10 13:16:58 nd1 multipathd: sdi path added to devmap 3600221900091530d0000da294afb6f6b
Dec 10 13:16:58 nd1 multipathd: dm-6: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-6: devmap already registered
Dec 10 13:16:58 nd1 multipathd: dm-9: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-8: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-7: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: 3600221900091530d0000da294afb6f6b: load table [0 473956352 multipath 1 queue_if_no_path 1 rdac 2 1 round-robin 0 2 1 8:96 600 8
Dec 10 13:16:58 nd1 multipathd: sdj path added to devmap 3600221900091530d0000da294afb6f6b
Dec 10 13:16:58 nd1 multipathd: dm-6: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-6: devmap already registered
Dec 10 13:16:58 nd1 multipathd: dm-9: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-8: add map (uevent)
Dec 10 13:16:58 nd1 multipathd: dm-7: add map (uevent)
Dec 10 13:16:59 nd1 multipathd: 3600221900092530000000e204afb71ee: load table [0 104857600 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 1 1 8:160 100]
Dec 10 13:16:59 nd1 multipathd: 3600221900092530000000e204afb71ee: event checker started
Dec 10 13:16:59 nd1 multipathd: sdk path added to devmap 3600221900092530000000e204afb71ee
Dec 10 13:16:59 nd1 multipathd: dm-7: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: 3600221900092530000000e204afb71ee: load table [0 104857600 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 2 1 8:160 100
Dec 10 13:17:00 nd1 multipathd: dm-7: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: dm-8: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: dm-9: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: dm-6: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: dm-6: devmap already registered
Dec 10 13:17:00 nd1 multipathd: 3600221900092530000000e204afb71ee: load table [0 104857600 multipath 1 queue_if_no_path 1 rdac 2 2 round-robin 0 1 1 8:208 400
Dec 10 13:17:00 nd1 multipathd: sdn path added to devmap 3600221900092530000000e204afb71ee
Dec 10 13:17:00 nd1 multipathd: 3600221900091530d0000da2b4afb700d: load table [0 1048576000 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 1 1 8:224 600
Dec 10 13:17:00 nd1 multipathd: 3600221900091530d0000da2b4afb700d: event checker started
Dec 10 13:17:00 nd1 multipathd: sdo path added to devmap 3600221900091530d0000da2b4afb700d
Dec 10 13:17:00 nd1 multipathd: dm-10: add map (uevent)
Dec 10 13:17:00 nd1 multipathd: dm-10: devmap already registered
Dec 10 13:17:01 nd1 multipathd: 3600221900092530000000e204afb71ee: load table [0 104857600 multipath 1 queue_if_no_path 1 rdac 2 1 round-robin 0 2 1 8:208 400
Dec 10 13:17:01 nd1 multipathd: sdm path added to devmap 3600221900092530000000e204afb71ee
Dec 10 13:17:01 nd1 multipathd: 3600221900091530d0000da2b4afb700d: load table [0 1048576000 multipath 1 queue_if_no_path 1 rdac 1 1 round-robin 0 2 1 8:224 600
Dec 10 13:17:01 nd1 multipathd: dm-10: add map (uevent)
Dec 10 13:17:01 nd1 multipathd: dm-10: devmap already registered
Dec 10 13:17:01 nd1 multipathd: dm-10: add map (uevent)
Dec 10 13:17:01 nd1 multipathd: dm-10: devmap already registered
Dec 10 13:17:01 nd1 multipathd: 3600221900091530d0000da2b4afb700d: load table [0 1048576000 multipath 1 queue_if_no_path 1 rdac 2 1 round-robin 0 2 1 8:224 600
Dec 10 13:17:01 nd1 multipathd: sdq path added to devmap 3600221900091530d0000da2b4afb700d
Dec 10 13:17:01 nd1 multipathd: dm-11: add map (uevent)
Dec 10 13:17:01 nd1 multipathd: dm-11: devmap already registered
Dec 10 13:17:02 nd1 multipathd: 3600221900092530000000e234afb72bc: event checker started
Dec 10 13:17:06 nd1 multipathd: 3600221900091530d00003fda49641d14: event checker started
Dec 10 13:17:06 nd1 multipathd: sdw path added to devmap 3600221900091530d00003fda49641d14
Dec 10 13:17:10 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: 65:96: mark as failed
Dec 10 13:17:13 nd1 multipathd: 65:96: reinstated
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:13 nd1 multipathd: dm-15: devmap already registered
Dec 10 13:17:14 nd1 logger: /etc/init.d/xdm: No changes for /etc/X11/xdm/Xservers
Dec 10 13:17:14 nd1 logger: /etc/init.d/xdm: No changes for /etc/X11/xdm/xdm-config
Dec 10 13:17:15 nd1 ntpd[8417]: ntpd 4.2.4p6 at 1.1549-o Fri May 8 08:34:30 UTC 2009 (1)
Dec 10 13:17:15 nd1 ntpd[8418]: precision = 1.000 usec
Dec 10 13:17:15 nd1 ntpd[8418]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #1 wildcard, ::#123 Disabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #2 lo, ::1#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #3 br0, fe80::21b:21ff:fe2b:7ef0#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #4 eth5, fe80::222:19ff:fe9f:6e7a#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #5 eth4, fe80::222:19ff:fe9f:6e78#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #6 lo, 127.0.0.1#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #7 lo, 127.0.0.2#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #8 eth4, 192.168.131.1#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #9 eth5, 192.168.130.1#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: Listening on interface #10 br0, 192.168.1.101#123 Enabled
Dec 10 13:17:15 nd1 ntpd[8418]: kernel time sync status 0040
Dec 10 13:17:15 nd1 ntpd[8418]: frequency initialized 38.889 PPM from /var/lib/ntp/drift/ntp.drift
Dec 10 13:17:18 nd1 smartd[8535]: smartd 5.39 2008-10-24 22:33 [x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce Allen, http://smartmontools.sourceforge.net
Dec 10 13:17:18 nd1 smartd[8535]: Opened configuration file /etc/smartd.conf
Dec 10 13:17:18 nd1 smartd[8535]: Drive: DEVICESCAN, implied '-a' Directive on line 26 of file /etc/smartd.conf
Dec 10 13:17:18 nd1 smartd[8535]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sda, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdb, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdb, NO MEDIUM present; skip device
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdc, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdc, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdc, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdd, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdd, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdd, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdd, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 sshd[8617]: Server listening on 0.0.0.0 port 22.
Dec 10 13:17:18 nd1 sshd[8617]: Server listening on :: port 22.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sde, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sde, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sde, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sde, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdf, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdf, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdf, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdf, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdg, opened
Dec 10 13:17:18 nd1 openais[8613]: [MAIN ] AIS Executive Service RELEASE 'subrev 1152 version 0.80'
Dec 10 13:17:18 nd1 openais[8613]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Dec 10 13:17:18 nd1 openais[8613]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Dec 10 13:17:18 nd1 openais[8613]: [MAIN ] AIS Executive Service: started and ready to provide service.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdg, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdg, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdg, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] Token Timeout (5000 ms) retransmit timeout (490 ms)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] token hold (382 ms) retransmits before loss (10 retrans)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] join (1000 ms) send_join (45 ms) consensus (2500 ms) merge (200 ms)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (20 messages)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] send threads (0 threads)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] RRP token expired timeout (490 ms)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] RRP token problem counter (2000 ms)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] RRP threshold (10 problem count)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] RRP mode set to none.
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] heartbeat_failures_allowed (0)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] max_network_delay (50 ms)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] The network interface [192.168.1.101] is now up.
Dec 10 13:17:18 nd1 /usr/sbin/cron[8647]: (CRON) STARTUP (V5.0)
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] Created or loaded sequence id 303204.192.168.1.101 for this ring.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdh, opened
Dec 10 13:17:18 nd1 openais[8613]: [TOTEM] entering GATHER state from 15.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdh, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdh, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdh, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdi, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdi, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdi, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdi, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdj, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdj, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdj, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdj, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdk, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdk, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdk, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdk, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdl, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdl, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdl, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdl, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdm, opened
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdm, does not support SMART Self-Test Log.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdm, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:18 nd1 smartd[8535]: Device: /dev/sdm, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdn, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdn, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdn, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdn, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdo, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdo, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdo, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdo, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: process_ais_conf: Reading configure
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: config_find_next: Processing additional logging options...
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: get_config_opt: Found 'off' for option: debug
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: get_config_opt: Found 'yes' for option: to_syslog
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: get_config_opt: Found 'daemon' for option: syslog_facility
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: config_find_next: Processing additional service options...
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: get_config_opt: Found 'yes' for option: use_logd
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: get_config_opt: Found 'yes' for option: use_mgmtd
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_plugin_init: CRM: Initialized
Dec 10 13:17:19 nd1 openais[8613]: [crm ] Logging: Initialized pcmk_plugin_init
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_plugin_init: Service: 9
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_plugin_init: Local node id: 1694607552
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_plugin_init: Local hostname: nd1
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: update_member: Creating entry for node 1694607552 born on 0
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: update_member: 0x73e6e0 Node 1694607552 now known as nd1 (was: (null))
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: update_member: Node nd1 now has 1 quorum votes (was 0)
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: update_member: Node 1694607552/nd1 is now: member
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8677 for process stonithd
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8678 for process cib
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8679 for process lrmd
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8682 for process attrd
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8683 for process pengine
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8684 for process crmd
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: spawn_child: Forked child 8685 for process mgmtd
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_startup: CRM: Initialized
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] Service initialized 'Pacemaker Cluster Manager'
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdp, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdp, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdp, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdp, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais extended virtual synchrony service'
Dec 10 13:17:19 nd1 lrmd: [8679]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 mgmtd: [8685]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 cib: [8678]: info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Dec 10 13:17:19 nd1 pengine: [8683]: info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Dec 10 13:17:19 nd1 attrd: [8682]: info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Dec 10 13:17:19 nd1 crmd: [8684]: info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Dec 10 13:17:19 nd1 stonithd: [8677]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 stonithd: [8677]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Dec 10 13:17:19 nd1 lrmd: [8679]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Dec 10 13:17:19 nd1 mgmtd: [8685]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Dec 10 13:17:19 nd1 cib: [8678]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 stonithd: [8677]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Dec 10 13:17:19 nd1 attrd: [8682]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 pengine: [8683]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 crmd: [8684]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Dec 10 13:17:19 nd1 attrd: [8682]: info: main: Starting up....
Dec 10 13:17:19 nd1 mgmtd: [8685]: debug: Enabling coredumps
Dec 10 13:17:19 nd1 attrd: [8682]: info: init_ais_connection: Creating connection to our AIS plugin
Dec 10 13:17:19 nd1 crmd: [8684]: info: main: CRM Hg Version: 0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Dec 10 13:17:19 nd1 cib: [8678]: info: G_main_add_TriggerHandler: Added signal manual handler
Dec 10 13:17:19 nd1 attrd: [8682]: info: init_ais_connection: AIS connection established
Dec 10 13:17:19 nd1 crmd: [8684]: info: crmd_init: Starting crmd
Dec 10 13:17:19 nd1 mgmtd: [8685]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Dec 10 13:17:19 nd1 stonithd: [8677]: info: init_ais_connection: Creating connection to our AIS plugin
Dec 10 13:17:19 nd1 attrd: [8682]: info: get_ais_nodeid: Server details: id=1694607552 uname=nd1
Dec 10 13:17:19 nd1 pengine: [8683]: info: main: Starting pengine
Dec 10 13:17:19 nd1 attrd: [8682]: info: crm_new_peer: Node nd1 now has id: 1694607552
Dec 10 13:17:19 nd1 cib: [8678]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Dec 10 13:17:19 nd1 mgmtd: [8685]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Dec 10 13:17:19 nd1 crmd: [8684]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Dec 10 13:17:19 nd1 lrmd: [8679]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Dec 10 13:17:19 nd1 lrmd: [8679]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Dec 10 13:17:19 nd1 attrd: [8682]: info: crm_new_peer: Node 1694607552 is now known as nd1
Dec 10 13:17:19 nd1 stonithd: [8677]: info: init_ais_connection: AIS connection established
Dec 10 13:17:19 nd1 cib: [8678]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Dec 10 13:17:19 nd1 stonithd: [8677]: info: get_ais_nodeid: Server details: id=1694607552 uname=nd1
Dec 10 13:17:19 nd1 lrmd: [8679]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Dec 10 13:17:19 nd1 stonithd: [8677]: info: crm_new_peer: Node nd1 now has id: 1694607552
Dec 10 13:17:19 nd1 stonithd: [8677]: info: crm_new_peer: Node 1694607552 is now known as nd1
Dec 10 13:17:19 nd1 lrmd: [8679]: info: Started.
Dec 10 13:17:19 nd1 stonithd: [8677]: notice: /usr/lib64/heartbeat/stonithd start up successfully.
Dec 10 13:17:19 nd1 stonithd: [8677]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Dec 10 13:17:19 nd1 mgmtd: [8685]: info: init_crm
Dec 10 13:17:19 nd1 mgmtd: [8685]: info: login to cib: 0, ret:-10
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais cluster membership service B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais availability management framework B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais checkpoint service B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais event service B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais distributed locking service B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais message service B.01.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais configuration service'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais cluster closed process group service v1.01'
Dec 10 13:17:19 nd1 openais[8613]: [SERV ] Service initialized 'openais cluster config database access v1.01'
Dec 10 13:17:19 nd1 openais[8613]: [SYNC ] Not using a virtual synchrony filter.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] Creating commit token because I am the rep.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] Saving state aru 0 high seq received 0
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] Storing new sequence id for ring 4a068
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] entering COMMIT state.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] entering RECOVERY state.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdq, opened
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] position [0] member 192.168.1.101:
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] previous ring seq 303204 rep 192.168.1.101
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] aru 0 high delivered 0 received flag 1
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] Did not need to originate any messages in recovery.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] Sending initial ORF token
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] CLM CONFIGURATION CHANGE
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] New Configuration:
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] Members Left:
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] Members Joined:
Dec 10 13:17:19 nd1 openais[8613]: [crm ] notice: pcmk_peer_update: Transitional membership event on ring 303208: memb=0, new=0, lost=0
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] CLM CONFIGURATION CHANGE
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] New Configuration:
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.101)
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] Members Left:
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] Members Joined:
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.101)
Dec 10 13:17:19 nd1 openais[8613]: [crm ] notice: pcmk_peer_update: Stable membership event on ring 303208: memb=1, new=1, lost=0
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_peer_update: NEW: nd1 1694607552
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_peer_update: MEMB: nd1 1694607552
Dec 10 13:17:19 nd1 openais[8613]: [MAIN ] info: update_member: Node nd1 now has process list: 00000000000000000000000000053312 (340754)
Dec 10 13:17:19 nd1 openais[8613]: [SYNC ] This node is within the primary component and will provide service.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] entering OPERATIONAL state.
Dec 10 13:17:19 nd1 openais[8613]: [CLM ] got nodejoin message 192.168.1.101
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_ipc: Recorded connection 0x7f59fc01d740 for attrd/8682
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_ipc: Recorded connection 0x7f59fc01dc50 for stonithd/8677
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdq, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdq, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdq, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdr, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdr, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdr, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdr, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sds, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sds, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sds, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sds, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 cib: [8678]: info: startCib: CIB Initialization completed successfully
Dec 10 13:17:19 nd1 cib: [8678]: info: init_ais_connection: Creating connection to our AIS plugin
Dec 10 13:17:19 nd1 cib: [8678]: info: init_ais_connection: AIS connection established
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_ipc: Recorded connection 0x7f59fc01cf90 for cib/8678
Dec 10 13:17:19 nd1 openais[8613]: [crm ] info: pcmk_ipc: Sending membership update 303208 to cib
Dec 10 13:17:19 nd1 cib: [8678]: info: get_ais_nodeid: Server details: id=1694607552 uname=nd1
Dec 10 13:17:19 nd1 cib: [8678]: info: crm_new_peer: Node nd1 now has id: 1694607552
Dec 10 13:17:19 nd1 cib: [8678]: info: crm_new_peer: Node 1694607552 is now known as nd1
Dec 10 13:17:19 nd1 cib: [8678]: info: cib_init: Starting cib mainloop
Dec 10 13:17:19 nd1 cib: [8678]: info: ais_dispatch: Membership 303208: quorum still lost
Dec 10 13:17:19 nd1 cib: [8678]: info: crm_update_peer: Node nd1: id=1694607552 state=member (new) addr=r(0) ip(192.168.1.101) (new) votes=1 (new) born=0 seen=303208 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdt, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdt, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdt, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdt, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdu, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdu, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdu, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdu, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 cib: [8758]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-79.raw
Dec 10 13:17:19 nd1 cib: [8758]: info: write_cib_contents: Wrote version 0.366.0 of the CIB to disk (digest: 576e61d6f8d8d3ebfb8a1445d6cb90a6)
Dec 10 13:17:19 nd1 cib: [8758]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.FlXGpK (digest: /var/lib/heartbeat/crm/cib.T73h8K)
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdv, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdv, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdv, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdv, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdw, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdw, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdw, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdw, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdx, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdx, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdx, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdx, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdy, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdy, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdy, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdy, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdz, opened
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdz, does not support SMART Self-Test Log.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdz, is SMART capable. Adding to "monitor" list.
Dec 10 13:17:19 nd1 smartd[8535]: Device: /dev/sdz, persistence not yet supported for SCSI; ignoring -s option.
Dec 10 13:17:19 nd1 smartd[8535]: Monitoring 0 ATA and 24 SCSI devices
Dec 10 13:17:19 nd1 smartd[8760]: smartd has fork()ed into background mode. New PID=8760.
Dec 10 13:17:19 nd1 openais[8613]: [TOTEM] entering GATHER state from 11.
Dec 10 13:17:20 nd1 xenstored: Checking store ...
Dec 10 13:17:20 nd1 xenstored: Checking store complete.
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:796: blktapctrl: v1.0.0
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [raw image (aio)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [raw image (sync)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [vmware image (vmdk)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [ramdisk image (ram)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [qcow disk (qcow)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [qcow2 disk (qcow2)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [ioemu disk]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl.c:798: Found driver: [raw image (cdrom)]
Dec 10 13:17:20 nd1 BLKTAPCTRL[8789]: blktapctrl_linux.c:23: /dev/xen/blktap0 device already exists
Dec 10 13:17:20 nd1 crmd: [8684]: info: do_cib_control: CIB connection established
Dec 10 13:17:20 nd1 crmd: [8684]: info: init_ais_connection: Creating connection to our AIS plugin
Dec 10 13:17:20 nd1 crmd: [8684]: info: init_ais_connection: AIS connection established
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_ipc: Recorded connection 0x7f59fc01d350 for crmd/8684
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_ipc: Sending membership update 303208 to crmd
Dec 10 13:17:20 nd1 crmd: [8684]: info: get_ais_nodeid: Server details: id=1694607552 uname=nd1
Dec 10 13:17:20 nd1 crmd: [8684]: info: crm_new_peer: Node nd1 now has id: 1694607552
Dec 10 13:17:20 nd1 crmd: [8684]: info: crm_new_peer: Node 1694607552 is now known as nd1
Dec 10 13:17:20 nd1 crmd: [8684]: info: do_ha_control: Connected to the cluster
Dec 10 13:17:20 nd1 crmd: [8684]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Dec 10 13:17:20 nd1 crmd: [8684]: info: crmd_init: Starting crmd's mainloop
Dec 10 13:17:20 nd1 crmd: [8684]: info: config_query_callback: Checking for expired actions every 900000ms
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: update_expected_votes: Expected quorum votes 1024 -> 3
Dec 10 13:17:20 nd1 crmd: [8684]: info: ais_dispatch: Membership 303208: quorum still lost
Dec 10 13:17:20 nd1 crmd: [8684]: info: crm_update_peer: Node nd1: id=1694607552 state=member (new) addr=r(0) ip(192.168.1.101) (new) votes=1 (new) born=0 seen=303208 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:20 nd1 crmd: [8684]: info: do_started: The local CRM is operational
Dec 10 13:17:20 nd1 crmd: [8684]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Dec 10 13:17:20 nd1 mgmtd: [8685]: debug: main: run the loop...
Dec 10 13:17:20 nd1 mgmtd: [8685]: info: Started.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] Creating commit token because I am the rep.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] Saving state aru a high seq received a
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] Storing new sequence id for ring 4a06c
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] entering COMMIT state.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] entering RECOVERY state.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] position [0] member 192.168.1.101:
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] previous ring seq 303208 rep 192.168.1.101
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] aru a high delivered a received flag 1
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] position [1] member 192.168.1.102:
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] previous ring seq 303208 rep 192.168.1.102
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] aru 58 high delivered 58 received flag 1
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] position [2] member 192.168.1.103:
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] previous ring seq 303208 rep 192.168.1.102
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] aru 58 high delivered 58 received flag 1
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] Did not need to originate any messages in recovery.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] Sending initial ORF token
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] CLM CONFIGURATION CHANGE
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] New Configuration:
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.101)
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] Members Left:
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] Members Joined:
Dec 10 13:17:20 nd1 openais[8613]: [crm ] notice: pcmk_peer_update: Transitional membership event on ring 303212: memb=1, new=0, lost=0
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: memb: nd1 1694607552
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] CLM CONFIGURATION CHANGE
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] New Configuration:
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.101)
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.102)
Dec 10 13:17:20 nd1 cib: [8678]: notice: ais_dispatch: Membership 303212: quorum aquired
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_new_peer: Node <null> now has id: 1711384768
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_update_peer: Node (null): id=1711384768 state=member (new) addr=r(0) ip(192.168.1.102) votes=0 born=0 seen=303212 proc=00000000000000000000000000000000
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_new_peer: Node <null> now has id: 1728161984
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_update_peer: Node (null): id=1728161984 state=member (new) addr=r(0) ip(192.168.1.103) votes=0 born=0 seen=303212 proc=00000000000000000000000000000000
Dec 10 13:17:20 nd1 cib: [8678]: info: ais_dispatch: Membership 303212: quorum retained
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_get_peer: Node 1711384768 is now known as nd2
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_update_peer: Node nd2: id=1711384768 state=member addr=r(0) ip(192.168.1.102) votes=1 (new) born=303208 seen=303212 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:20 nd1 cib: [8678]: info: ais_dispatch: Membership 303212: quorum retained
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_get_peer: Node 1728161984 is now known as nd3
Dec 10 13:17:20 nd1 cib: [8678]: info: crm_update_peer: Node nd3: id=1728161984 state=member addr=r(0) ip(192.168.1.103) votes=1 (new) born=303208 seen=303212 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.103)
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] Members Left:
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] Members Joined:
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.102)
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] r(0) ip(192.168.1.103)
Dec 10 13:17:20 nd1 openais[8613]: [crm ] notice: pcmk_peer_update: Stable membership event on ring 303212: memb=3, new=2, lost=0
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Creating entry for node 1711384768 born on 303212
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node 1711384768/unknown is now: member
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: NEW: .pending. 1711384768
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Creating entry for node 1728161984 born on 303212
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node 1728161984/unknown is now: member
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: NEW: .pending. 1728161984
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: MEMB: nd1 1694607552
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: MEMB: .pending. 1711384768
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: pcmk_peer_update: MEMB: .pending. 1728161984
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: send_member_notification: Sending membership update 303212 to 2 children
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: 0x73e6e0 Node 1694607552 ((null)) born on: 303212
Dec 10 13:17:20 nd1 openais[8613]: [SYNC ] This node is within the primary component and will provide service.
Dec 10 13:17:20 nd1 openais[8613]: [TOTEM] entering OPERATIONAL state.
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: 0x7f59fc01d690 Node 1711384768 (nd2) born on: 303208
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: 0x7f59fc01d690 Node 1711384768 now known as nd2 (was: (null))
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node nd2 now has process list: 00000000000000000000000000053312 (340754)
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node nd2 now has 1 quorum votes (was 0)
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: send_member_notification: Sending membership update 303212 to 2 children
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: 0x7f59fc01d450 Node 1728161984 (nd3) born on: 303208
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: 0x7f59fc01d450 Node 1728161984 now known as nd3 (was: (null))
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node nd3 now has process list: 00000000000000000000000000053312 (340754)
Dec 10 13:17:20 nd1 openais[8613]: [MAIN ] info: update_member: Node nd3 now has 1 quorum votes (was 0)
Dec 10 13:17:20 nd1 openais[8613]: [crm ] info: send_member_notification: Sending membership update 303212 to 2 children
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] got nodejoin message 192.168.1.101
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] got nodejoin message 192.168.1.102
Dec 10 13:17:20 nd1 openais[8613]: [CLM ] got nodejoin message 192.168.1.103
Dec 10 13:17:21 nd1 cib: [8678]: info: cib_process_diff: Diff 0.367.8 -> 0.367.9 not applied to 0.366.0: current "epoch" is less than required
Dec 10 13:17:21 nd1 cib: [8678]: info: cib_server_process_diff: Requesting re-sync from peer
Dec 10 13:17:21 nd1 cib: [8678]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 68): 0.0.0 (Application of an update diff failed, requesting a full refresh)
Dec 10 13:17:21 nd1 cib: [8678]: info: cib_replace_notify: Replaced: 0.366.0 -> 0.367.9 from nd3
Dec 10 13:17:21 nd1 cib: [8815]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-80.raw
Dec 10 13:17:21 nd1 cib: [8815]: info: write_cib_contents: Wrote version 0.367.0 of the CIB to disk (digest: ba446b0ee34e686a10e996d09d7ef1b1)
Dec 10 13:17:21 nd1 cib: [8815]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.c3H0Xi (digest: /var/lib/heartbeat/crm/cib.zlPQ5n)
Dec 10 13:17:21 nd1 crmd: [8684]: info: ais_dispatch: Membership 303208: quorum still lost
Dec 10 13:17:21 nd1 crmd: [8684]: notice: ais_dispatch: Membership 303212: quorum aquired
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_new_peer: Node <null> now has id: 1711384768
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_update_peer: Node (null): id=1711384768 state=member (new) addr=r(0) ip(192.168.1.102) votes=0 born=0 seen=303212 proc=00000000000000000000000000000000
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_new_peer: Node <null> now has id: 1728161984
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_update_peer: Node (null): id=1728161984 state=member (new) addr=r(0) ip(192.168.1.103) votes=0 born=0 seen=303212 proc=00000000000000000000000000000000
Dec 10 13:17:21 nd1 crmd: [8684]: info: ais_dispatch: Membership 303212: quorum retained
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_get_peer: Node 1711384768 is now known as nd2
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_update_peer: Node nd2: id=1711384768 state=member addr=r(0) ip(192.168.1.102) votes=1 (new) born=303208 seen=303212 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:21 nd1 crmd: [8684]: info: ais_dispatch: Membership 303212: quorum retained
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_get_peer: Node 1728161984 is now known as nd3
Dec 10 13:17:21 nd1 crmd: [8684]: info: crm_update_peer: Node nd3: id=1728161984 state=member addr=r(0) ip(192.168.1.103) votes=1 (new) born=303208 seen=303212 proc=00000000000000000000000000053312 (new)
Dec 10 13:17:21 nd1 crmd: [8684]: info: update_dc: Set DC to nd3 (3.0.1)
Dec 10 13:17:21 nd1 crmd: [8684]: info: update_attrd: Connecting to attrd...
Dec 10 13:17:21 nd1 crmd: [8684]: info: update_attrd: Updating terminate=<none> via attrd for nd1
Dec 10 13:17:21 nd1 crmd: [8684]: info: update_attrd: Updating shutdown=<none> via attrd for nd1
Dec 10 13:17:21 nd1 crmd: [8684]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Dec 10 13:17:21 nd1 crmd: [8684]: info: erase_xpath_callback: Deletion of "//node_state[@uname='nd1']/transient_attributes": ok (rc=0)
Dec 10 13:17:21 nd1 crmd: [8684]: info: erase_xpath_callback: Deletion of "//node_state[@uname='nd1']/lrm": ok (rc=0)
Dec 10 13:17:23 nd1 /usr/sbin/cron[9022]: (CRON) STARTUP (V5.0)
Dec 10 13:17:29 nd1 attrd: [8682]: info: main: Sending full refresh
Dec 10 13:17:29 nd1 attrd: [8682]: info: main: Starting mainloop...
Dec 10 13:17:29 nd1 attrd: [8682]: info: crm_new_peer: Node nd3 now has id: 1728161984
Dec 10 13:17:29 nd1 attrd: [8682]: info: crm_new_peer: Node 1728161984 is now known as nd3
Dec 10 13:17:29 nd1 attrd: [8682]: info: find_hash_entry: Creating hash entry for terminate
Dec 10 13:17:29 nd1 attrd: [8682]: info: find_hash_entry: Creating hash entry for shutdown
Dec 10 13:17:29 nd1 attrd: [8682]: info: crm_new_peer: Node nd2 now has id: 1711384768
Dec 10 13:17:29 nd1 attrd: [8682]: info: crm_new_peer: Node 1711384768 is now known as nd2
Dec 10 13:17:29 nd1 attrd: [8682]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Dec 10 13:17:29 nd1 attrd: [8682]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate
Dec 10 13:17:29 nd1 attrd: [8682]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown
Dec 10 13:18:10 nd1 sshd[9084]: Accepted keyboard-interactive/pam for root from 192.168.1.168 port 4407 ssh2
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=4:3:7:a1c27862-c5ca-487d-858d-a1cb5f382849 op=dlm:0_monitor_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:dlm:0: monitor
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=5:3:7:a1c27862-c5ca-487d-858d-a1cb5f382849 op=o2cb:0_monitor_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:o2cb:0: monitor
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=6:3:7:a1c27862-c5ca-487d-858d-a1cb5f382849 op=clvm:0_monitor_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:clvm:0: monitor
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation clvm:0_monitor_0 (call=4, rc=7, cib-update=8, confirmed=true) complete not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:monitor:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:monitor:stderr) 2009/12/10_13:18:18 INFO: configfs not laoded
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation o2cb:0_monitor_0 (call=3, rc=7, cib-update=9, confirmed=true) complete not running
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation dlm:0_monitor_0 (call=2, rc=7, cib-update=10, confirmed=true) complete not running
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=12:3:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=dlm:0_start_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:dlm:0: start
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:start:stderr) FATAL: module '/lib/modules/2.6.27.39-0.3-xen/kernel/fs/dlm/dlm.ko' is unsupported Use --allow-unsupported or set allow_unsupported_modules to 1 in /etc/modprobe.d/unsupported-modules
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:start:stderr) 2009/12/10_13:18:18 ERROR: /sys/kernel/config/dlm not available
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation dlm:0_start_0 (call=5, rc=0, cib-update=14, confirmed=true) complete ok
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=13:3:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=dlm:0_monitor_120000 )
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=22:3:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=o2cb:0_start_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:o2cb:0: start
Dec 10 13:18:18 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=32:3:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=clvm:0_start_0 )
Dec 10 13:18:18 nd1 lrmd: [8679]: info: rsc:clvm:0: start
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (clvm:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (clvm:0:start:stderr) 2009/12/10_13:18:18 INFO: Starting clvm:0
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation dlm:0_monitor_120000 (call=6, rc=7, cib-update=15, confirmed=false) complete not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) 2009/12/10_13:18:18 INFO: Stack glue driver not loaded
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) 2009/12/10_13:18:18 INFO: Starting o2cb:0
Dec 10 13:18:18 nd1 modprobe: WARNING: module '/lib/modules/2.6.27.39-0.3-xen/kernel/fs/dlm/dlm.ko' is unsupported Use --allow-unsupported or set allow_unsupported_modules to 1 in /etc/modprobe.d/unsupported-modules
Dec 10 13:18:18 nd1 clvmd: Unable to create lockspace for CLVM: Success
Dec 10 13:18:18 nd1 modprobe: FATAL: Error inserting ocfs2_stack_user (/lib/modules/2.6.27.39-0.3-xen/weak-updates/updates/ocfs2/ocfs2_stack_user.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (clvm:0:start:stderr) clvmd could not connect to cluster manager Consult syslog for more information
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (o2cb:0:start:stderr) 2009/12/10_13:18:18 ERROR: Could not load ocfs2_stack_user
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (clvm:0:start:stderr) logd is not running
Dec 10 13:18:18 nd1 lrmd: [8679]: info: RA output: (clvm:0:start:stderr) 2009/12/10_13:18:18 ERROR: Could not start /usr/sbin/clvmd
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation o2cb:0_start_0 (call=7, rc=5, cib-update=16, confirmed=true) complete not installed
Dec 10 13:18:18 nd1 crmd: [8684]: info: process_lrm_event: LRM operation clvm:0_start_0 (call=8, rc=1, cib-update=17, confirmed=true) complete unknown error
Dec 10 13:18:19 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=5:4:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=o2cb:0_stop_0 )
Dec 10 13:18:19 nd1 lrmd: [8679]: info: rsc:o2cb:0: stop
Dec 10 13:18:19 nd1 crmd: [8684]: info: do_lrm_rsc_op: Performing key=4:4:0:a1c27862-c5ca-487d-858d-a1cb5f382849 op=clvm:0_stop_0 )
Dec 10 13:18:19 nd1 lrmd: [8679]: info: rsc:clvm:0: stop
Dec 10 13:18:19 nd1 crmd: [8684]: info: process_lrm_event: LRM operation clvm:0_stop_0 (call=10, rc=0, cib-update=18, confirmed=true) complete ok
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) logd is not running
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) 2009/12/10_13:18:19 ERROR: Wrong stack
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) logd is not running
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) 2009/12/10_13:18:19 INFO: Stopping o2cb:0
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) logd is not running
Dec 10 13:18:19 nd1 lrmd: [8679]: info: RA output: (o2cb:0:stop:stderr) 2009/12/10_13:18:19 ERROR: Unable to unload modules: the cluster is still online
Dec 10 13:18:19 nd1 crmd: [8684]: info: process_lrm_event: LRM operation o2cb:0_stop_0 (call=9, rc=1, cib-update=19, confirmed=true) complete unknown error
Dec 10 13:18:22 nd1 stonithd: [8677]: info: crm_new_peer: Node nd3 now has id: 1728161984
Dec 10 13:18:22 nd1 stonithd: [8677]: info: crm_new_peer: Node 1728161984 is now known as nd3
Dec 10 13:18:26 nd1 mgmtd: [8685]: info: CIB query: cib
Dec 10 13:18:32 nd1 mgmtd: [8685]: ERROR: unpack_rsc_op: Hard error - o2cb:2_start_0 failed with rc=5: Preventing o2cb-clone from re-starting on nd3
Dec 10 13:18:32 nd1 mgmtd: [8685]: ERROR: unpack_rsc_op: Hard error - o2cb:0_start_0 failed with rc=5: Preventing o2cb-clone from re-starting on nd1
Dec 10 13:19:19 nd1 stonithd: [8677]: WARN: Performed 1 more non-realtime malloc calls.
Dec 10 13:19:19 nd1 stonithd: [8677]: info: Total non-realtime malloc bytes: 880640
Dec 10 13:20:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:22:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:24:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:26:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:27:19 nd1 cib: [8678]: info: cib_stats: Processed 83 operations (120.00us average, 0% utilization) in the last 10min
Dec 10 13:28:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:30:01 nd1 /usr/sbin/cron[9977]: (root) CMD (/usr/lib/SMT/bin/smt-repeated-register)
Dec 10 13:30:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:32:05 nd1 clvmd: Unable to create lockspace for CLVM: Success
Dec 10 13:32:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:34:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:36:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:38:18 nd1 lrmd: [8679]: info: RA output: (dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Dec 10 13:40:13 nd1 sshd[10093]: Accepted keyboard-interactive/pam for root from 192.168.1.168 port 4637 ssh2
Dec 10 13:40:13 nd1 sshd[10093]: subsystem request for sftp
-------------- next part --------------
<cib validate-with="pacemaker-1.0" crm_feature_set="3.0.1" have-quorum="1" admin_epoch="0" epoch="367" num_updates="48" cib-last-written="Thu Dec 10 13:02:30 2009" dc-uuid="nd3">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.0.3-0080ec086ae9c20ad5c4c3562000c0ad68374f0a"/>
<nvpair id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes" value="3"/>
<nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1260440569"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="nd1" uname="nd1" type="normal"/>
<node id="nd2" uname="nd2" type="normal"/>
<node id="nd3" uname="nd3" type="normal"/>
</nodes>
<resources>
<clone id="dlm-clone">
<meta_attributes id="dlm-clone-meta_attributes">
<nvpair id="nvpair-e36fb7de-1e1e-460b-94eb-49e3b68de045" name="clone-max" value="3"/>
<nvpair id="nvpair-e7165899-ba8c-48a8-9138-cc5236c19717" name="target-role" value="started"/>
<nvpair id="nvpair-34e724a7-76cf-4dd9-82ab-d7756a68c142" name="globally-unique" value="false"/>
<nvpair id="nvpair-c404080e-dc1d-4ac7-9901-a1406750c17b" name="interleave" value="true"/>
</meta_attributes>
<primitive class="ocf" id="dlm" provider="pacemaker" type="controld">
<operations id="dlm-operations">
<op id="dlm-op-monitor-10" interval="120" name="monitor" start-delay="0" timeout="240"/>
</operations>
</primitive>
</clone>
<clone id="o2cb-clone">
<meta_attributes id="o2cb-clone-meta_attributes">
<nvpair id="nvpair-aedbc563-94f3-4e1d-ba6b-fa1aec7c3dde" name="clone-max" value="3"/>
<nvpair id="nvpair-685746f7-0e7a-4ce1-8018-628b300d5113" name="target-role" value="started"/>
<nvpair id="nvpair-996ea224-6192-4e50-ab91-ad1f898d6430" name="globally-unique" value="false"/>
<nvpair id="nvpair-f22b45f9-ca12-48d6-9d2e-2aa08f7db494" name="interleave" value="true"/>
</meta_attributes>
<primitive class="ocf" id="o2cb" provider="ocfs2" type="o2cb">
<operations id="o2cb-operations">
<op id="o2cb-op-monitor-10" interval="120" name="monitor" timeout="240"/>
</operations>
</primitive>
</clone>
<clone id="clvm-clone">
<meta_attributes id="clvm-clone-meta_attributes">
<nvpair id="nvpair-d3e52e22-06f4-4f32-ae7d-fc5916bd50d6" name="clone-max" value="3"/>
<nvpair id="nvpair-8382d2b6-cf0a-4a99-881e-23752f5ad1b3" name="target-role" value="started"/>
<nvpair id="nvpair-b09a2eeb-3d2e-4a3c-ac06-c80fd5c05226" name="interleave" value="true"/>
<nvpair id="nvpair-ed0ed128-4c5a-4ee8-b0b4-97e90c8e1940" name="ordered" value="true"/>
</meta_attributes>
<primitive class="ocf" id="clvm" provider="lvm2" type="clvmd">
<operations id="clvm-operations">
<op id="clvm-op-monitor-10" interval="120" name="monitor" timeout="240"/>
</operations>
<instance_attributes id="clvm-instance_attributes">
<nvpair id="nvpair-5fe5091f-9187-4643-b2d6-ec806ae433d6" name="daemon_timeout" value="30"/>
</instance_attributes>
</primitive>
</clone>
</resources>
<constraints>
<rsc_colocation id="o2cb-with-dlm" rsc="dlm-clone" score="INFINITY" with-rsc="o2cb-clone"/>
<rsc_order first="dlm-clone" id="start-o2cb-after-dlm" then="o2cb-clone"/>
<rsc_colocation id="colo-dlm-clvm" rsc="dlm-clone" score="INFINITY" with-rsc="clvm-clone"/>
<rsc_order first="dlm-clone" id="start-clvm-after-dlm" then="clvm-clone"/>
</constraints>
</configuration>
/cib>
More information about the Pacemaker
mailing list