Mar 24 10:56:15 ha-test2 cib: [4823]: info: cib_stats: Processed 128 operations (234.00us average, 0% utilization) in the last 10min
Mar 24 11:06:15 ha-test2 cib: [4823]: info: cib_stats: Processed 32 operations (625.00us average, 0% utilization) in the last 10min
Mar 24 11:16:15 ha-test2 cib: [4823]: info: cib_stats: Processed 32 operations (0.00us average, 0% utilization) in the last 10min
Mar 24 11:19:28 ha-test2 crm_shadow: [10214]: info: Invoked: crm_shadow 
Mar 24 11:19:28 ha-test2 crm_attribute: [10219]: info: Invoked: crm_attribute -N ha-test1 -n standby -v on -l forever 
Mar 24 11:19:28 ha-test2 cib: [10220]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-47.raw
Mar 24 11:19:28 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=82:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:28 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:20: notify
Mar 24 11:19:28 ha-test2 cib: [10220]: info: write_cib_contents: Wrote version 0.131.0 of the CIB to disk (digest: 21d74a7a136d837dc73311b025e9f44f)
Mar 24 11:19:28 ha-test2 cib: [10220]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.yGKN9r (digest: /var/lib/heartbeat/crm/cib.NKBdFL)
Mar 24 11:19:28 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=20, rc=0, cib-update=27, confirmed=true) ok
Mar 24 11:19:29 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=83:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:29 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:21: notify
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=21, rc=0, cib-update=28, confirmed=true) ok
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=76:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:22: notify
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=22, rc=0, cib-update=29, confirmed=true) ok
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=77:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:23: notify
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=23, rc=0, cib-update=30, confirmed=true) ok
Mar 24 11:19:36 ha-test2 crm_shadow: [10349]: info: Invoked: crm_shadow 
Mar 24 11:19:38 ha-test2 crm_shadow: [10379]: info: Invoked: crm_shadow 
Mar 24 11:19:38 ha-test2 crm_attribute: [10383]: info: Invoked: crm_attribute -N ha-test1 -n standby -v off -l forever 
Mar 24 11:19:38 ha-test2 cib: [10384]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-48.raw
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=70:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:24: notify
Mar 24 11:19:38 ha-test2 cib: [10384]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: c431886c0decf8058af7150f21a06f18)
Mar 24 11:19:38 ha-test2 cib: [10384]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.aUCFt0 (digest: /var/lib/heartbeat/crm/cib.F5gXiS)
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=24, rc=0, cib-update=31, confirmed=true) ok
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=71:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:25: notify
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=25, rc=0, cib-update=32, confirmed=true) ok
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=83:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:26: notify
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=26, rc=0, cib-update=33, confirmed=true) ok
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=84:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:27: notify
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=27, rc=0, cib-update=34, confirmed=true) ok
Mar 24 11:22:42 ha-test2 corosync[4811]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:22:42 ha-test2 corosync[4811]:   [TOTEM ] The network interface is down.
Mar 24 11:23:12 ha-test2 cib: [1496]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: a6e332c373698e925ac0aa9d889cecf0)
Mar 24 11:23:12 ha-test2 cib: [1496]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.VJP89B (digest: /var/lib/heartbeat/crm/cib.anww6n)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_cib_control: CIB connection established
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: init_ais_connection: AIS connection established
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x8ac65d8 for crmd/1492
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_ipc: Sending membership update 848 to (null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_ha_control: Connected to the cluster
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crmd_init: Starting crmd's mainloop
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: Delaying start, Config not read (0000000000000040)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: config_query_callback: Checking for expired actions every 900000ms
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: The local CRM is operational
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: update_expected_votes: Expected quorum votes 1024 -> 2
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:14 ha-test2 cib: [1488]: info: cib_replace_notify: Replaced: 0.132.0 -> 0.132.25 from ha-test1
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: update_attrd: Connecting to attrd...
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for terminate
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for shutdown
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test2 logd: [1858]: info: logd started with default configuration.
Mar 24 11:23:14 ha-test2 logd: [1858]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:23:14 ha-test2 logd: [1862]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:23:14 ha-test2 cib: [1855]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-50.raw
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for master-resDRBD:0
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:0=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for probe_complete
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation probe_complete=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for master-resDRBD:1
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=<null>: cib not connected
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=9:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:2: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=10:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resDLM:0:3: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=11:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resO2CB:0:4: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=12:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resFS:0:5: monitor
Mar 24 11:23:15 ha-test2 Filesystem[1891]: WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resFS:0:monitor:stderr) 2010/03/24_11:23:15 WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDLM:0:monitor:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_monitor_0 (call=3, rc=7, cib-update=8, confirmed=true) not running
Mar 24 11:23:15 ha-test2 cib: [1855]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: 075394cb7bbc83fedfbbd40279948d19)
Mar 24 11:23:15 ha-test2 o2cb[1890]: INFO: configfs not laoded
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:monitor:stderr) 2010/03/24_11:23:15 INFO: configfs not laoded
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true) not running
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_monitor_0 (call=5, rc=7, cib-update=10, confirmed=true) not running
Mar 24 11:23:15 ha-test2 cib: [1855]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.qy4I1I (digest: /var/lib/heartbeat/crm/cib.4lRGPB)
Mar 24 11:23:15 ha-test2 crm_attribute: [2031]: info: Invoked: crm_attribute -N ha-test2 -n master-resDRBD:1 -l reboot -D 
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_0 (call=2, rc=7, cib-update=11, confirmed=true) not running
Mar 24 11:23:15 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:15 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation probe_complete=true: cib not connected
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=17:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_start_0 )
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:6: start
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout)         allow-two-primaries;
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.33.1/kernel/drivers/crypto/padlock-sha.ko): No such device
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (1000)
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=1000: cib not connected
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_start_0 (call=6, rc=0, cib-update=12, confirmed=true) ok
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=79:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:7: notify
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=7, rc=0, cib-update=13, confirmed=true) ok
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=18:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_30000 )
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, rc=0, cib-update=14, confirmed=false) ok
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: cib_connect: Connected to the CIB after 1 signon attempts
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: cib_connect: Sending full refresh
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_perform_update: Sent update 4: probe_complete=true
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_perform_update: Sent update 10: master-resDRBD:1=10000
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: send_direct_ack: ACK'ing resource op resDRBD:1_monitor_30000 from 5:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269426197-4
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=87:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:9: notify
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=43:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_start_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDLM:0:10: start
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDLM:0:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=9, rc=0, cib-update=16, confirmed=true) ok
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=17:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_promote_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:11: promote
Mar 24 11:23:17 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 2711 (0x8abfd60)
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:promote:stdout)         allow-two-primaries;
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:promote:stdout) 
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_promote_0 (call=11, rc=0, cib-update=17, confirmed=true) ok
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=88:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:12: notify
Mar 24 11:23:18 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=12, rc=0, cib-update=18, confirmed=true) ok
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=18:14:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_20000 )
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_20000 (call=13, rc=8, cib-update=19, confirmed=false) master
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_start_0 (call=10, rc=0, cib-update=20, confirmed=true) ok
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=44:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_120000 )
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=51:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_start_0 )
Mar 24 11:23:18 ha-test2 lrmd: [1489]: info: rsc:resO2CB:0:15: start
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_monitor_120000 (call=14, rc=0, cib-update=21, confirmed=false) ok
Mar 24 11:23:19 ha-test2 o2cb[3091]: INFO: Stack glue driver not loaded
Mar 24 11:23:19 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:23:19 INFO: Stack glue driver not loaded
Mar 24 11:23:19 ha-test2 o2cb[3091]: INFO: Starting resO2CB:0
Mar 24 11:23:19 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:23:19 INFO: Starting resO2CB:0
Mar 24 11:23:19 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3160 (0x8ac0fb8)
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_start_0 (call=15, rc=0, cib-update=22, confirmed=true) ok
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=52:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_120000 )
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=59:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_start_0 )
Mar 24 11:23:21 ha-test2 lrmd: [1489]: info: rsc:resFS:0:17: start
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_monitor_120000 (call=16, rc=0, cib-update=23, confirmed=false) ok
Mar 24 11:23:21 ha-test2 Filesystem[3811]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:23:21 ha-test2 lrmd: [1489]: info: RA output: (resFS:0:start:stderr) 2010/03/24_11:23:21 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_start_0 (call=17, rc=0, cib-update=24, confirmed=true) ok
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=60:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_120000 )
Mar 24 11:23:22 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_monitor_120000 (call=18, rc=0, cib-update=25, confirmed=false) ok
Mar 24 11:25:22 ha-test2 cibadmin: [4425]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:25:22 ha-test2 root: Mark:HB_REPORT:1269426320
Mar 24 11:25:25 ha-test2 cibadmin: [4632]: info: Invoked: cibadmin -Ql 
Mar 24 11:25:25 ha-test2 crm_shadow: [4672]: info: Invoked: crm_shadow 
Mar 24 11:25:25 ha-test2 cibadmin: [4673]: info: Invoked: cibadmin -Ql 
Mar 24 11:28:54 ha-test2 cibadmin: [5558]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:28:54 ha-test2 root: Mark:HB_REPORT:1269426533
Mar 24 11:28:56 ha-test2 cibadmin: [5761]: info: Invoked: cibadmin -Ql 
Mar 24 11:28:56 ha-test2 crm_shadow: [5801]: info: Invoked: crm_shadow 
Mar 24 11:28:56 ha-test2 cibadmin: [5802]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:25 ha-test2 cibadmin: [6385]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:29:27 ha-test2 root: Mark:HB_REPORT:1269426564
Mar 24 11:29:29 ha-test2 cibadmin: [6579]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:29 ha-test2 crm_shadow: [6619]: info: Invoked: crm_shadow 
Mar 24 11:29:29 ha-test2 cibadmin: [6620]: info: Invoked: cibadmin -Ql 
Mar 24 11:33:12 ha-test2 cib: [1488]: info: cib_stats: Processed 122 operations (573.00us average, 0% utilization) in the last 10min
Mar 24 11:35:22 ha-test2 cibadmin: [7890]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:35:26 ha-test2 cibadmin: [7970]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:35:26 ha-test2 root: Mark:HB_REPORT:1269426925
Mar 24 11:35:28 ha-test2 cibadmin: [8265]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:28 ha-test2 crm_shadow: [8305]: info: Invoked: crm_shadow 
Mar 24 11:35:28 ha-test2 cibadmin: [8306]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:42 ha-test2 cibadmin: [9253]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:38:42 ha-test2 root: Mark:HB_REPORT:1269427121
Mar 24 11:38:44 ha-test2 cibadmin: [9548]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:45 ha-test2 crm_shadow: [9588]: info: Invoked: crm_shadow 
Mar 24 11:38:45 ha-test2 cibadmin: [9589]: info: Invoked: cibadmin -Ql 
Mar 24 11:43:12 ha-test2 cib: [1488]: info: cib_stats: Processed 46 operations (869.00us average, 0% utilization) in the last 10min
Mar 24 11:53:12 ha-test2 cib: [1488]: info: cib_stats: Processed 32 operations (0.00us average, 0% utilization) in the last 10min
Mar 24 11:56:17 ha-test2 crm_shadow: [12243]: info: Invoked: crm_shadow 
Mar 24 11:56:17 ha-test2 crm_attribute: [12248]: info: Invoked: crm_attribute -N ha-test1 -n standby -v on -l forever 
Mar 24 11:56:17 ha-test2 cib: [12249]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-51.raw
Mar 24 11:56:17 ha-test2 cib: [12249]: info: write_cib_contents: Wrote version 0.133.0 of the CIB to disk (digest: 68d6db75d4871405c3ab119cbd61ae10)
Mar 24 11:56:17 ha-test2 cib: [12249]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.rwPUa0 (digest: /var/lib/heartbeat/crm/cib.yIn479)
Mar 24 11:56:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=86:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:19: notify
Mar 24 11:56:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=19, rc=0, cib-update=26, confirmed=true) ok
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=87:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:20: notify
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=20, rc=0, cib-update=27, confirmed=true) ok
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=80:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:21: notify
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=21, rc=0, cib-update=28, confirmed=true) ok
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=81:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:22: notify
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=22, rc=0, cib-update=29, confirmed=true) ok
Mar 24 11:56:25 ha-test2 crm_shadow: [12404]: info: Invoked: crm_shadow 
Mar 24 11:56:25 ha-test2 crm_attribute: [12408]: info: Invoked: crm_attribute -N ha-test1 -n standby -v off -l forever 
Mar 24 11:56:25 ha-test2 cib: [12409]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-52.raw
Mar 24 11:56:25 ha-test2 cib: [12409]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: 25c404d549614fcfa017a9957e21fa94)
Mar 24 11:56:25 ha-test2 cib: [12409]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.EggLDs (digest: /var/lib/heartbeat/crm/cib.XXgL34)
Mar 24 11:56:25 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=70:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:25 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:23: notify
Mar 24 11:56:25 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:25 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=23, rc=0, cib-update=30, confirmed=true) ok
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=71:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:24: notify
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=24, rc=0, cib-update=31, confirmed=true) ok
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=83:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:25: notify
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=25, rc=0, cib-update=32, confirmed=true) ok
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=84:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:26: notify
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=26, rc=0, cib-update=33, confirmed=true) ok
Mar 24 11:56:52 ha-test2 corosync[1466]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:56:52 ha-test2 corosync[1466]:   [TOTEM ] The network interface is down.
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] The network interface [10.133.99.92] is now up.
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: process_ais_conf: Reading configure
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_init: Local handle: 2730409743423111170 for logging
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_next: Processing additional logging options...
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Found 'off' for option: debug
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'off' for option: to_file
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Found 'daemon' for option: syslog_facility
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_init: Local handle: 5880381755227111427 for service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_next: Processing additional service options...
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: CRM: Initialized
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] Logging: Initialized pcmk_startup
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Maximum core file size is: 4294967295
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Service: 9
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Local hostname: ha-test2
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_update_nodeid: Local node id: 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Creating entry for node 1550025994 born on 0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96b9f98 Node 1550025994 now known as ha-test2 (was: (null))
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test2 now has 1 quorum votes (was 0)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1493 for process stonithd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1494 for process cib
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1495 for process lrmd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1496 for process attrd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1497 for process pengine
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1498 for process crmd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.0.7
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: Started.
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: Stack hogger failed 0xffffffff
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: Invoked: /usr/lib/heartbeat/attrd 
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Starting up
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 cib: [1494]: info: Invoked: /usr/lib/heartbeat/cib 
Mar 24 11:57:18 ha-test2 cib: [1494]: info: G_main_add_TriggerHandler: Added signal manual handler
Mar 24 11:57:18 ha-test2 cib: [1494]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 cib: [1494]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Mar 24 11:57:18 ha-test2 pengine: [1497]: info: Invoked: /usr/lib/heartbeat/pengine 
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Mar 24 11:57:18 ha-test2 pengine: [1497]: info: main: Starting pengine
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync configuration service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Mar 24 11:57:18 ha-test2 cib: [1494]: info: startCib: CIB Initialization completed successfully
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 cib: [1494]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync profile loading service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 852: memb=0, new=0, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 852: memb=1, new=1, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test2 now has process list: 00000000000000000000000000013312 (78610)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test2 corosync[1460]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 cib: [1494]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c0350 for attrd/1496
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c0658 for cib/1494
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 cib: [1494]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Sending membership update 852 to (null)
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Cluster connection active
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_init: Starting cib mainloop
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Accepting attribute updates
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Starting mainloop...
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000000354(null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: Invoked: /usr/lib/heartbeat/crmd 
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: main: CRM Hg Version: 54d7869bfe3691eb723b1d47810e5585d8246b58
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crmd_init: Starting crmd
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_cib_control: CIB connection established
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c2560 for crmd/1498
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Sending membership update 852 to (null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_ha_control: Connected to the cluster
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crmd_init: Starting crmd's mainloop
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000000354(null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: Delaying start, Config not read (0000000000000040)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: config_query_callback: Checking for expired actions every 900000ms
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_expected_votes: Expected quorum votes 1024 -> 2
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: The local CRM is operational
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c2cf0 for stonithd/1493
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 stonithd: [1493]: notice: /usr/lib/heartbeat/stonithd start up successfully.
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 cib: [1501]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-53.raw
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 856: memb=1, new=0, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 856: memb=2, new=1, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Creating entry for node 1533248778 born on 856
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node 1533248778/unknown is now: member
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: NEW:  .pending. 1533248778
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: .pending. 1533248778
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96b9f98 Node 1550025994 ((null)) born on: 856
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node <null> now has id: 1533248778
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node (null): id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=0 born=0 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96c2a30 Node 1533248778 (ha-test1) born on: 840
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96c2a30 Node 1533248778 now known as ha-test1 (was: (null))
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test1 now has process list: 00000000000000000000000000013312 (78610)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test1 now has 1 quorum votes (was 0)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_get_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member addr=r(0) ip(10.133.99.91)  votes=1 (new) born=840 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_process_diff: Diff 0.134.22 -> 0.134.23 not applied to 0.134.0: current "num_updates" is less than required
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_server_process_diff: Requesting re-sync from peer
Mar 24 11:57:18 ha-test2 corosync[1460]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 139): 0.0.0 (Application of an update diff failed, requesting a full refresh)
Mar 24 11:57:18 ha-test2 cib: [1501]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: cc1d42d0a4fb39a8b5cc16b326f29393)
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_server_process_diff: Not applying diff 0.134.23 -> 0.134.24 (sync in progress)
Mar 24 11:57:18 ha-test2 cib: [1501]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.NDF4ks (digest: /var/lib/heartbeat/crm/cib.iKqto0)
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_server_process_diff: Not applying diff 0.134.24 -> 0.134.25 (sync in progress)
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_replace_notify: Replaced: 0.134.0 -> 0.134.25 from ha-test1
Mar 24 11:57:18 ha-test2 cib: [1513]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-54.raw
Mar 24 11:57:18 ha-test2 cib: [1513]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: 100caec81dbb156797a7fce9abd1d580)
Mar 24 11:57:18 ha-test2 cib: [1513]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.taBEFs (digest: /var/lib/heartbeat/crm/cib.jTdD30)
Mar 24 11:57:19 ha-test2 logd: [1854]: info: logd started with default configuration.
Mar 24 11:57:19 ha-test2 logd: [1854]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:19 ha-test2 logd: [1857]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_new_peer: Node <null> now has id: 1533248778
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_update_peer: Node (null): id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=0 born=0 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_get_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member addr=r(0) ip(10.133.99.91)  votes=1 (new) born=840 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: update_attrd: Connecting to attrd...
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for terminate
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for shutdown
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for master-resDRBD:0
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:0=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for probe_complete
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation probe_complete=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for master-resDRBD:1
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=<null>: cib not connected
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=9:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:2: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=10:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resDLM:0:3: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=11:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resO2CB:0:4: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=12:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resFS:0:5: monitor
Mar 24 11:57:19 ha-test2 o2cb[1950]: INFO: configfs not laoded
Mar 24 11:57:19 ha-test2 Filesystem[1951]: WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:monitor:stderr) 2010/03/24_11:57:19 INFO: configfs not laoded
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true) not running
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resFS:0:monitor:stderr) 2010/03/24_11:57:19 WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resDLM:0:monitor:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_monitor_0 (call=3, rc=7, cib-update=10, confirmed=true) not running
Mar 24 11:57:20 ha-test2 crm_attribute: [2062]: info: Invoked: crm_attribute -N ha-test2 -n master-resDRBD:1 -l reboot -D 
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_0 (call=2, rc=7, cib-update=11, confirmed=true) not running
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_monitor_0 (call=5, rc=7, cib-update=12, confirmed=true) not running
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation probe_complete=true: cib not connected
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=17:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_start_0 )
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:6: start
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout)         allow-two-primaries;
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.33.1/kernel/drivers/crypto/padlock-sha.ko): No such device
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_start_0 (call=6, rc=0, cib-update=13, confirmed=true) ok
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=79:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:7: notify
Mar 24 11:57:21 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=7, rc=0, cib-update=14, confirmed=true) ok
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=18:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_30000 )
Mar 24 11:57:21 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:21 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, rc=0, cib-update=15, confirmed=false) ok
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: cib_connect: Connected to the CIB after 1 signon attempts
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: cib_connect: Sending full refresh
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_perform_update: Sent update 4: probe_complete=true
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_perform_update: Sent update 10: master-resDRBD:1=10000
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: send_direct_ack: ACK'ing resource op resDRBD:1_monitor_30000 from 5:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269428243-6
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=87:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:9: notify
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=43:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_start_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDLM:0:10: start
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDLM:0:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=9, rc=0, cib-update=17, confirmed=true) ok
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=17:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_promote_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:11: promote
Mar 24 11:57:23 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3372 (0x96c3628)
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:promote:stdout)         allow-two-primaries;
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:promote:stdout) 
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_promote_0 (call=11, rc=0, cib-update=18, confirmed=true) ok
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=88:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:12: notify
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=12, rc=0, cib-update=19, confirmed=true) ok
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=18:24:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_20000 )
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_20000 (call=13, rc=8, cib-update=20, confirmed=false) master
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_start_0 (call=10, rc=0, cib-update=21, confirmed=true) ok
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=44:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_120000 )
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=51:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_start_0 )
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: rsc:resO2CB:0:15: start
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_monitor_120000 (call=14, rc=0, cib-update=22, confirmed=false) ok
Mar 24 11:57:24 ha-test2 o2cb[3704]: INFO: Stack glue driver not loaded
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:57:24 INFO: Stack glue driver not loaded
Mar 24 11:57:24 ha-test2 o2cb[3704]: INFO: Starting resO2CB:0
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:57:24 INFO: Starting resO2CB:0
Mar 24 11:57:25 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3758 (0x96c8648)
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_start_0 (call=15, rc=0, cib-update=23, confirmed=true) ok
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=52:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_120000 )
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=59:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_start_0 )
Mar 24 11:57:27 ha-test2 lrmd: [1495]: info: rsc:resFS:0:17: start
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_monitor_120000 (call=16, rc=0, cib-update=24, confirmed=false) ok
Mar 24 11:57:27 ha-test2 Filesystem[3979]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:57:27 ha-test2 lrmd: [1495]: info: RA output: (resFS:0:start:stderr) 2010/03/24_11:57:27 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_start_0 (call=17, rc=0, cib-update=25, confirmed=true) ok
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=60:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_120000 )
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_monitor_120000 (call=18, rc=0, cib-update=26, confirmed=false) ok
Mar 24 11:59:02 ha-test2 cibadmin: [4292]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:59:03 ha-test2 root: Mark:HB_REPORT:1269428341
