Nov 13 13:44:17 vm1 corosync[15863]:   [MAIN  ] main.c:main:1171 Corosync Cluster Engine ('2.3.2.7-a911'): started and ready to provide service.
Nov 13 13:44:17 vm1 corosync[15863]:   [MAIN  ] main.c:main:1172 Corosync built-in features: watchdog upstart snmp pie relro bindnow
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemnet.c:totemnet_instance_initialize:242 Initializing transport (UDP/IP Multicast).
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemcrypto.c:init_nss:579 Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemnet.c:totemnet_instance_initialize:242 Initializing transport (UDP/IP Multicast).
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemcrypto.c:init_nss:579 Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemudp.c:timer_function_netif_check_timeout:670 The network interface [192.168.101.141] is now up.
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync configuration map access [0]
Nov 13 13:44:17 vm1 corosync[15865]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cmap
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync configuration service [1]
Nov 13 13:44:17 vm1 corosync[15865]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cfg
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync cluster closed process group service v1.01 [2]
Nov 13 13:44:17 vm1 corosync[15865]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cpg
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync profile loading service [4]
Nov 13 13:44:17 vm1 corosync[15865]:   [WD    ] wd.c:setup_watchdog:631 No Watchdog, try modprobe <a watchdog>
Nov 13 13:44:17 vm1 corosync[15865]:   [WD    ] wd.c:wd_scan_resources:580 no resources configured.
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync watchdog service [7]
Nov 13 13:44:17 vm1 corosync[15865]:   [QUORUM] vsf_quorum.c:quorum_exec_init_fn:274 Using quorum provider corosync_votequorum
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync vote quorum service v1.0 [5]
Nov 13 13:44:17 vm1 corosync[15865]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: votequorum
Nov 13 13:44:17 vm1 corosync[15865]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync cluster quorum service v0.1 [3]
Nov 13 13:44:17 vm1 corosync[15865]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: quorum
Nov 13 13:44:17 vm1 corosync[15865]:   [TOTEM ] totemudp.c:timer_function_netif_check_timeout:670 The network interface [192.168.102.141] is now up.
Nov 13 13:44:18 vm1 corosync[15865]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.141:4) was formed. Members joined: -1062705779
Nov 13 13:44:18 vm1 corosync[15865]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[1]: -1062705779
Nov 13 13:44:18 vm1 corosync[15865]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:18 vm1 corosync[15865]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.141:8) was formed. Members joined: -1062705778
Nov 13 13:44:18 vm1 corosync[15865]:   [QUORUM] vsf_quorum.c:quorum_api_set_quorum:148 This node is within the primary component and will provide service.
Nov 13 13:44:18 vm1 corosync[15865]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[2]: -1062705779 -1062705778
Nov 13 13:44:18 vm1 corosync[15865]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:18 vm1 corosync[15865]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.141:12) was formed. Members joined: -1062705777
Nov 13 13:44:18 vm1 corosync[15865]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[3]: -1062705779 -1062705778 -1062705777
Nov 13 13:44:18 vm1 corosync[15865]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 pacemakerd[15874]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_ipc_connect: Could not establish pacemakerd connection: Connection refused (111)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: get_cluster_type: Detected an active 'corosync' cluster
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: mcp_read_config: Reading configure for stack: corosync
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: mcp_read_config: Configured corosync to accept connections from group 492: OK (1)
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: main: Starting Pacemaker 1.1.10 (Build: 2383f6c):  ncurses libqb-logging libqb-ipc lha-fencing nagios  corosync-native snmp
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: main: Maximum core file size is: 18446744073709551615
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: qb_ipcs_us_publish: server name: pacemakerd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Created entry 47e23e4e-8d89-44d7-ade8-b727d086f719/0xff9030 for node (null)/3232261517 (1 total)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: cluster_connect_quorum: Quorum acquired
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Using uid=496 and group=492 for process cib
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15878 for process cib
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15879 for process stonith-ng
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15880 for process lrmd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Using uid=496 and group=492 for process attrd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15881 for process attrd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Using uid=496 and group=492 for process pengine
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15882 for process pengine
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Using uid=496 and group=492 for process crmd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: start_child: Forked child 15883 for process crmd
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: main: Starting mainloop
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: pcmk_quorum_notification: Membership 12: quorum retained (3)
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node vm1[3232261517] - state is now member (was (null))
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Created entry 4b1694e5-d501-4f98-a4b3-82e2729ae396/0x10fa8e0 for node (null)/3232261518 (2 total)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261518
Nov 13 13:44:20 vm1 attrd[15881]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 attrd[15881]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm1 cib[15878]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 cib[15878]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm1 lrmd[15880]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 stonith-ng[15879]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 stonith-ng[15879]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 attrd[15881]:     info: main: Starting up
Nov 13 13:44:20 vm1 attrd[15881]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm1 attrd[15881]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm1 attrd[15881]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm1 lrmd[15880]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 lrmd[15880]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm1 lrmd[15880]:     info: qb_ipcs_us_publish: server name: lrmd
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm1 cib[15878]:   notice: main: Using new config location: /var/lib/pacemaker/cib
Nov 13 13:44:20 vm1 cib[15878]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm1 cib[15878]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm1 cib[15878]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
Nov 13 13:44:20 vm1 cib[15878]:  warning: retrieveCib: Cluster configuration not found: /var/lib/pacemaker/cib/cib.xml
Nov 13 13:44:20 vm1 cib[15878]:  warning: readCibXmlFile: Primary configuration corrupt or unusable, trying backups in /var/lib/pacemaker/cib
Nov 13 13:44:20 vm1 cib[15878]:  warning: readCibXmlFile: Continuing with an empty configuration.
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm1 stonith-ng[15879]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm1 lrmd[15880]:     info: main: Starting
Nov 13 13:44:20 vm1 cib[15878]:     info: validate_with_relaxng: Creating RNG parser context
Nov 13 13:44:20 vm1 pengine[15882]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 pengine[15882]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 pengine[15882]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm1 crmd[15883]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm1 crmd[15883]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm1 pengine[15882]:     info: qb_ipcs_us_publish: server name: pengine
Nov 13 13:44:20 vm1 pengine[15882]:     info: main: Starting pengine
Nov 13 13:44:20 vm1 crmd[15883]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm1 crmd[15883]:   notice: main: CRM Git Version: 2383f6c
Nov 13 13:44:20 vm1 crmd[15883]:     info: do_log: FSA: Input I_STARTUP from crmd_init() received in state S_STARTING
Nov 13 13:44:20 vm1 crmd[15883]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm1 crmd[15883]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm1 crmd[15883]:     info: crm_ipc_connect: Could not establish cib_shm connection: Connection refused (111)
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261518] - state is now member (was (null))
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Created entry 8c58deab-c20e-41d3-b0da-8ece9c0ac2f1/0x10fabe0 for node (null)/3232261519 (3 total)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261519
Nov 13 13:44:20 vm1 cib[15878]:     info: startCib: CIB Initialization completed successfully
Nov 13 13:44:20 vm1 cib[15878]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_get_peer: Created entry 4a155865-3b20-4c1b-92ef-fc18f238821f/0x1fed130 for node (null)/3232261517 (1 total)
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:20 vm1 attrd[15881]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261517] - state is now member (was (null))
Nov 13 13:44:20 vm1 attrd[15881]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_get_peer: Created entry c18fea60-cb54-4d98-abd8-87fca3b361ad/0x19436a0 for node (null)/3232261517 (1 total)
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261519
Nov 13 13:44:20 vm1 pacemakerd[15874]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_get_peer: Created entry 53f2667c-1243-48e0-b350-5b1413d2c3ab/0x1352360 for node (null)/3232261517 (1 total)
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:20 vm1 cib[15878]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm1 attrd[15881]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:20 vm1 attrd[15881]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:20 vm1 attrd[15881]:     info: main: Cluster connection active
Nov 13 13:44:20 vm1 attrd[15881]:     info: qb_ipcs_us_publish: server name: attrd
Nov 13 13:44:20 vm1 attrd[15881]:     info: main: Accepting attribute updates
Nov 13 13:44:20 vm1 attrd[15881]:     info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:20 vm1 stonith-ng[15879]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:20 vm1 stonith-ng[15879]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:20 vm1 stonith-ng[15879]:     info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
Nov 13 13:44:20 vm1 pacemakerd[15874]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:20 vm1 cib[15878]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:20 vm1 cib[15878]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:20 vm1 cib[15878]:     info: qb_ipcs_us_publish: server name: cib_ro
Nov 13 13:44:20 vm1 cib[15878]:     info: qb_ipcs_us_publish: server name: cib_rw
Nov 13 13:44:20 vm1 cib[15878]:     info: qb_ipcs_us_publish: server name: cib_shm
Nov 13 13:44:20 vm1 cib[15878]:     info: cib_init: Starting cib mainloop
Nov 13 13:44:20 vm1 cib[15878]:     info: pcmk_cpg_membership: Joined[0.0] cib.3232261517 
Nov 13 13:44:20 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[0.0] cib.3232261517 
Nov 13 13:44:20 vm1 cib[15884]:     info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: 978cb58a57d1ff0f3e53e793331143d7)
Nov 13 13:44:20 vm1 cib[15884]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.rjUedC (digest: /var/lib/pacemaker/cib/cib.Ceepd4)
Nov 13 13:44:20 vm1 cib[15878]:     info: pcmk_cpg_membership: Joined[1.0] cib.3232261518 
Nov 13 13:44:20 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[1.0] cib.3232261517 
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_get_peer: Created entry ed2a0024-eb7a-4964-8d99-8ae499b14693/0x13552a0 for node (null)/3232261518 (2 total)
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[1.1] cib.3232261518 
Nov 13 13:44:20 vm1 cib[15878]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:21 vm1 cib[15878]:     info: pcmk_cpg_membership: Joined[2.0] cib.3232261519 
Nov 13 13:44:21 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[2.0] cib.3232261517 
Nov 13 13:44:21 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[2.1] cib.3232261518 
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_get_peer: Created entry 1be75549-bb16-43ef-8f89-0ae7866f5052/0x1355310 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm1 cib[15878]:     info: pcmk_cpg_membership: Member[2.2] cib.3232261519 
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_client_new: Connecting 0x1355380 for uid=496 gid=492 pid=15883 id=7bc8a5ad-4bdd-48b2-981b-bfd88861945f
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_cib_control: CIB connection established
Nov 13 13:44:21 vm1 crmd[15883]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:21 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.0.0)
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Created entry 21dfed2a-ea12-4608-98b4-45fc01561e11/0x1414c50 for node (null)/3232261517 (1 total)
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:21 vm1 crmd[15883]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:21 vm1 crmd[15883]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:21 vm1 crmd[15883]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:21 vm1 crmd[15883]:     info: peer_update_callback: vm1 is now (null)
Nov 13 13:44:21 vm1 crmd[15883]:   notice: cluster_connect_quorum: Quorum acquired
Nov 13 13:44:21 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/3, version=0.0.0)
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_ha_control: Connected to the cluster
Nov 13 13:44:21 vm1 crmd[15883]:     info: lrmd_ipc_connect: Connecting to lrmd
Nov 13 13:44:21 vm1 lrmd[15880]:     info: crm_client_new: Connecting 0x1b86c10 for uid=496 gid=492 pid=15883 id=b88cb348-8886-42a5-bb3d-9ea70cadc946
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_lrm_control: LRM connection established
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_started: Delaying start, no membership data (0000000000100000)
Nov 13 13:44:21 vm1 crmd[15883]:     info: pcmk_quorum_notification: Membership 12: quorum retained (3)
Nov 13 13:44:21 vm1 crmd[15883]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node vm1[3232261517] - state is now member (was (null))
Nov 13 13:44:21 vm1 crmd[15883]:     info: peer_update_callback: vm1 is now member (was (null))
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Created entry e2a0f72f-5a96-4feb-ab42-b7ebcea911d0/0x155bdb0 for node (null)/3232261518 (2 total)
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:21 vm1 crmd[15883]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261518
Nov 13 13:44:21 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/4, version=0.0.0)
Nov 13 13:44:21 vm1 crmd[15883]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:21 vm1 crmd[15883]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261518] - state is now member (was (null))
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Created entry cb5f37c1-f59c-4417-88eb-1a18addbcd1a/0x15598e0 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm1 crmd[15883]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261519
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_client_new: Connecting 0x11a3be0 for uid=496 gid=492 pid=15881 id=7dadebe1-bb78-4c38-bc59-37c0c7c29b82
Nov 13 13:44:21 vm1 attrd[15881]:     info: attrd_cib_connect: Connected to the CIB after 2 attempts
Nov 13 13:44:21 vm1 attrd[15881]:     info: main: CIB connection active
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Joined[0.0] attrd.3232261517 
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[0.0] attrd.3232261517 
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Joined[1.0] attrd.3232261518 
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[1.0] attrd.3232261517 
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_get_peer: Created entry c3446606-a271-488b-a38a-a1269f3c03ad/0x1ff2f00 for node (null)/3232261518 (2 total)
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[1.1] attrd.3232261518 
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:21 vm1 attrd[15881]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261518] - state is now member (was (null))
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Joined[2.0] attrd.3232261519 
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[2.0] attrd.3232261517 
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[2.1] attrd.3232261518 
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_get_peer: Created entry c8bbef81-9b84-4c05-b435-f946b89cb77d/0x1ff2f70 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm1 attrd[15881]:     info: pcmk_cpg_membership: Member[2.2] attrd.3232261519 
Nov 13 13:44:21 vm1 attrd[15881]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:21 vm1 attrd[15881]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:21 vm1 cib[15878]:     info: crm_client_new: Connecting 0x13d6370 for uid=0 gid=0 pid=15879 id=6b4f4e55-9d9b-4e29-aa80-b9806663187b
Nov 13 13:44:21 vm1 stonith-ng[15879]:   notice: setup_cib: Watching for stonith topology changes
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: qb_ipcs_us_publish: server name: stonith-ng
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: main: Starting stonith-ng mainloop
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Joined[0.0] stonith-ng.3232261517 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[0.0] stonith-ng.3232261517 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Joined[1.0] stonith-ng.3232261518 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[1.0] stonith-ng.3232261517 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_get_peer: Created entry ed84558a-74e8-4b26-b508-558824c85888/0x1947790 for node (null)/3232261518 (2 total)
Nov 13 13:44:21 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.0.0)
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[1.1] stonith-ng.3232261518 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:21 vm1 crmd[15883]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261519
Nov 13 13:44:21 vm1 crmd[15883]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:21 vm1 stonith-ng[15879]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:21 vm1 stonith-ng[15879]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: init_cib_cache_cb: Updating device list from the cib: init
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: unpack_nodes: Creating a fake local node
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Joined[2.0] stonith-ng.3232261519 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[2.0] stonith-ng.3232261517 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[2.1] stonith-ng.3232261518 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_get_peer: Created entry 0e7ba1e2-ac96-414f-8f91-447bb3876068/0x1948a50 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: pcmk_cpg_membership: Member[2.2] stonith-ng.3232261519 
Nov 13 13:44:21 vm1 stonith-ng[15879]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:21 vm1 crmd[15883]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:21 vm1 crmd[15883]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_started: Delaying start, Config not read (0000000000000040)
Nov 13 13:44:21 vm1 crmd[15883]:     info: qb_ipcs_us_publish: server name: crmd
Nov 13 13:44:21 vm1 crmd[15883]:   notice: do_started: The local CRM is operational
Nov 13 13:44:21 vm1 crmd[15883]:     info: do_log: FSA: Input I_PENDING from do_started() received in state S_STARTING
Nov 13 13:44:21 vm1 crmd[15883]:   notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Nov 13 13:44:21 vm1 cib[15878]:     info: cib_process_request: Completed cib_slave operation for section 'all': OK (rc=0, origin=local/crmd/5, version=0.0.0)
Nov 13 13:44:22 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:22 vm1 stonith-ng[15879]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Joined[0.0] crmd.3232261517 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[0.0] crmd.3232261517 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Joined[1.0] crmd.3232261518 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[1.0] crmd.3232261517 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[1.1] crmd.3232261518 
Nov 13 13:44:22 vm1 crmd[15883]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Joined[2.0] crmd.3232261519 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[2.0] crmd.3232261517 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[2.1] crmd.3232261518 
Nov 13 13:44:22 vm1 crmd[15883]:     info: pcmk_cpg_membership: Member[2.2] crmd.3232261519 
Nov 13 13:44:22 vm1 crmd[15883]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:22 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:22 vm1 crmd[15883]:     info: peer_update_callback: vm2 is now member
Nov 13 13:44:22 vm1 crmd[15883]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:22 vm1 crmd[15883]:     info: peer_update_callback: vm3 is now member
Nov 13 13:44:23 vm1 cib[15878]:     info: crm_client_new: Connecting 0x13d68d0 for uid=0 gid=0 pid=14287 id=7e66a1ab-3f5c-494d-ace7-2c089552d090
Nov 13 13:44:23 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/5, version=0.0.0)
Nov 13 13:44:23 vm1 stonith-ng[15879]:     info: crm_client_new: Connecting 0x194ab50 for uid=496 gid=492 pid=15883 id=e6cedc3f-e233-4a66-9291-ce359bd76aad
Nov 13 13:44:23 vm1 stonith-ng[15879]:     info: stonith_command: Processed register from crmd.15883: OK (0)
Nov 13 13:44:23 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify from crmd.15883: OK (0)
Nov 13 13:44:23 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify from crmd.15883: OK (0)
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
Nov 13 13:44:42 vm1 crmd[15883]:  warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:44:42 vm1 crmd[15883]:     info: election_timer_cb: Election election-0 complete
Nov 13 13:44:42 vm1 crmd[15883]:     info: election_timeout_popped: Election failed: Declaring ourselves the winner
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_log: FSA: Input I_ELECTION_DC from election_timeout_popped() received in state S_ELECTION
Nov 13 13:44:42 vm1 crmd[15883]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped ]
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_te_control: Registering TE UUID: 154fb289-24e8-407e-9a03-69a510480b60
Nov 13 13:44:42 vm1 crmd[15883]:     info: set_graph_functions: Setting custom graph functions
Nov 13 13:44:42 vm1 pengine[15882]:     info: crm_client_new: Connecting 0x1caa6f0 for uid=496 gid=492 pid=15883 id=c9c42a36-df9d-47e0-a006-f20302ba50d7
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_dc_takeover: Taking over DC status for this partition
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_readwrite: We are now in R/W mode
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/6, version=0.0.0)
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/7, version=0.0.1)
Nov 13 13:44:42 vm1 cib[15878]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:42 vm1 cib[15878]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: No such device or address (rc=-6, origin=local/crmd/8, version=0.0.1)
Nov 13 13:44:42 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.1.1
Nov 13 13:44:42 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="0" num_updates="1"/>
Nov 13 13:44:42 vm1 cib[15878]:   notice: cib:diff: ++       <cluster_property_set id="cib-bootstrap-options">
Nov 13 13:44:42 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-2383f6c"/>
Nov 13 13:44:42 vm1 cib[15878]:   notice: cib:diff: ++       </cluster_property_set>
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/9, version=0.1.1)
Nov 13 13:44:42 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: No such device or address (rc=-6, origin=local/crmd/10, version=0.1.1)
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: Making join offers based on membership 12
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-1: Sending offer to vm3
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm3[3232261519] - join-1 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-1: Sending offer to vm1
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm1[3232261517] - join-1 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-1: Sending offer to vm2
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm2[3232261518] - join-1 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_dc_join_offer_all: join-1: Waiting on 3 outstanding join acks
Nov 13 13:44:42 vm1 crmd[15883]:  warning: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm3[3232261519] - join-2 phase 1 -> 0
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm1[3232261517] - join-2 phase 1 -> 0
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm2[3232261518] - join-2 phase 1 -> 0
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-2: Sending offer to vm3
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm3[3232261519] - join-2 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-2: Sending offer to vm1
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm1[3232261517] - join-2 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: join_make_offer: join-2: Sending offer to vm2
Nov 13 13:44:42 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm2[3232261518] - join-2 phase 0 -> 1
Nov 13 13:44:42 vm1 crmd[15883]:     info: do_dc_join_offer_all: join-2: Waiting on 3 outstanding join acks
Nov 13 13:44:43 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.2.1
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="1" num_updates="1"/>
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/11, version=0.2.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/12, version=0.2.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/13, version=0.2.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/14, version=0.2.1)
Nov 13 13:44:43 vm1 crmd[15883]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/15, version=0.2.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/16, version=0.2.1)
Nov 13 13:44:43 vm1 cib[15892]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw
Nov 13 13:44:43 vm1 cib[15892]:     info: write_cib_contents: Wrote version 0.2.0 of the CIB to disk (digest: 7c397f6c57041145e23f3494e809aec1)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm1[3232261517] - join-2 phase 1 -> 2
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_expected: do_dc_join_filter_offer: Node vm1[3232261517] - expected state is now member (was (null))
Nov 13 13:44:43 vm1 cib[15892]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.LJIZDB (digest: /var/lib/pacemaker/cib/cib.f4K371)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm2[3232261518] - join-2 phase 1 -> 2
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_expected: do_dc_join_filter_offer: Node vm2[3232261518] - expected state is now member (was (null))
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm3[3232261519] - join-2 phase 1 -> 2
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_expected: do_dc_join_filter_offer: Node vm3[3232261519] - expected state is now member (was (null))
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Nov 13 13:44:43 vm1 crmd[15883]:     info: crmd_join_phase_log: join-2: vm3=integrated
Nov 13 13:44:43 vm1 crmd[15883]:     info: crmd_join_phase_log: join-2: vm1=integrated
Nov 13 13:44:43 vm1 crmd[15883]:     info: crmd_join_phase_log: join-2: vm2=integrated
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_dc_join_finalize: join-2: Syncing our CIB to the rest of the cluster
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/17, version=0.2.1)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm3[3232261519] - join-2 phase 2 -> 3
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm1[3232261517] - join-2 phase 2 -> 3
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm2[3232261518] - join-2 phase 2 -> 3
Nov 13 13:44:43 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.3.1
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="2" num_updates="1"/>
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: ++       <node id="3232261519" uname="vm3"/>
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/18, version=0.3.1)
Nov 13 13:44:43 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.4.1
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="3" num_updates="1"/>
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: ++       <node id="3232261517" uname="vm1"/>
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/19, version=0.4.1)
Nov 13 13:44:43 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm1']/transient_attributes
Nov 13 13:44:43 vm1 crmd[15883]:     info: update_attrd_helper: Connecting to attrd... 5 retries remaining
Nov 13 13:44:43 vm1 attrd[15881]:     info: crm_client_new: Connecting 0x1ff0000 for uid=496 gid=492 pid=15883 id=144c4790-c0bd-4d4b-b789-d7c2462cf699
Nov 13 13:44:43 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.5.1
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="4" num_updates="1"/>
Nov 13 13:44:43 vm1 cib[15878]:   notice: cib:diff: ++       <node id="3232261518" uname="vm2"/>
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/20, version=0.5.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm1']/transient_attributes: OK (rc=0, origin=local/crmd/21, version=0.5.1)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm3[3232261519] - join-2 phase 3 -> 4
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_dc_join_ack: join-2: Updating node state to member for vm3
Nov 13 13:44:43 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm3']/lrm
Nov 13 13:44:43 vm1 attrd[15881]:     info: attrd_client_message: Starting an election to determine the writer
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm3']/lrm: OK (rc=0, origin=local/crmd/22, version=0.5.1)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/23, version=0.5.2)
Nov 13 13:44:43 vm1 cib[15893]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw
Nov 13 13:44:43 vm1 cib[15893]:     info: write_cib_contents: Wrote version 0.5.0 of the CIB to disk (digest: 630d79f602055b52fd2ea79fdbd1baf8)
Nov 13 13:44:43 vm1 cib[15893]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.cazqnI (digest: /var/lib/pacemaker/cib/cib.gBFD48)
Nov 13 13:44:43 vm1 attrd[15881]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:43 vm1 attrd[15881]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:43 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting terminate[vm1] = (null)
Nov 13 13:44:43 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting shutdown[vm1] = (null)
Nov 13 13:44:43 vm1 cib[15878]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm3']/transient_attributes: OK (rc=0, origin=vm3/crmd/10, version=0.5.2)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm1[3232261517] - join-2 phase 3 -> 4
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_dc_join_ack: join-2: Updating node state to member for vm1
Nov 13 13:44:43 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm1']/lrm
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm1']/lrm: OK (rc=0, origin=local/crmd/24, version=0.5.2)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/25, version=0.5.3)
Nov 13 13:44:43 vm1 cib[15878]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm2']/transient_attributes: OK (rc=0, origin=vm2/crmd/10, version=0.5.3)
Nov 13 13:44:43 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm2[3232261518] - join-2 phase 3 -> 4
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_dc_join_ack: join-2: Updating node state to member for vm2
Nov 13 13:44:43 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm2']/lrm
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm2']/lrm: OK (rc=0, origin=local/crmd/26, version=0.5.3)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/27, version=0.5.4)
Nov 13 13:44:43 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:43 vm1 attrd[15881]:     info: election_count_vote: Election 1 (owner: 3232261519) pass: vote from vm3 (Uptime)
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Nov 13 13:44:43 vm1 crmd[15883]:     info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/28, version=0.5.4)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/29, version=0.5.4)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/30, version=0.5.5)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/31, version=0.5.5)
Nov 13 13:44:43 vm1 pengine[15882]:    error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Nov 13 13:44:43 vm1 pengine[15882]:    error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Nov 13 13:44:43 vm1 pengine[15882]:    error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:44:43 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:44:43 vm1 pengine[15882]:   notice: stage6: Delaying fencing operations until there are resources to manage
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1384317883-15) derived from /var/lib/pacemaker/pengine/pe-input-0.bz2
Nov 13 13:44:43 vm1 pengine[15882]:   notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-0.bz2
Nov 13 13:44:43 vm1 pengine[15882]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Nov 13 13:44:43 vm1 attrd[15881]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:43 vm1 attrd[15881]:     info: election_count_vote: Election 1 (owner: 3232261518) lost: vote from vm2 (Uptime)
Nov 13 13:44:43 vm1 attrd[15881]:     info: election_count_vote: Election 2 (owner: 3232261518) lost: vote from vm2 (Uptime)
Nov 13 13:44:43 vm1 attrd[15881]:     info: election_count_vote: Election 3 (owner: 3232261518) lost: vote from vm2 (Uptime)
Nov 13 13:44:43 vm1 attrd[15881]:     info: election_count_vote: Election 4 (owner: 3232261518) lost: vote from vm2 (Uptime)
Nov 13 13:44:43 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 4: probe_complete probe_complete on vm3 - no waiting
Nov 13 13:44:43 vm1 crmd[15883]:     info: te_rsc_command: Action 4 confirmed - no wait
Nov 13 13:44:43 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 3: probe_complete probe_complete on vm2 - no waiting
Nov 13 13:44:43 vm1 crmd[15883]:     info: te_rsc_command: Action 3 confirmed - no wait
Nov 13 13:44:43 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 2: probe_complete probe_complete on vm1 (local) - no waiting
Nov 13 13:44:43 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting probe_complete[vm1] = true
Nov 13 13:44:43 vm1 crmd[15883]:     info: te_rsc_command: Action 2 confirmed - no wait
Nov 13 13:44:43 vm1 crmd[15883]:   notice: run_graph: Transition 0 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-0.bz2): Complete
Nov 13 13:44:43 vm1 crmd[15883]:     info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Nov 13 13:44:43 vm1 crmd[15883]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 13 13:44:43 vm1 attrd[15881]:   notice: attrd_peer_message: Processing sync-response from vm2
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/2, version=0.5.6)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/3, version=0.5.6)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/4, version=0.5.7)
Nov 13 13:44:43 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/5, version=0.5.8)
Nov 13 13:44:51 vm1 crmd[15883]:     info: throttle_send_command: Updated throttle state to 0000
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_new: Connecting 0x15625a0 for uid=0 gid=0 pid=15903 id=12e28600-5dcd-41e5-a2c8-2462c7696cf8
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.5.8)
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_destroy: Destroying 0 events
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_new: Connecting 0x15625a0 for uid=0 gid=0 pid=15904 id=b8c3d466-bf55-4b35-a3e8-6129e139d346
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.5.8)
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_destroy: Destroying 0 events
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_new: Connecting 0x15625a0 for uid=0 gid=0 pid=15930 id=7c4f8a2a-0e8f-4fee-a29b-ec5d3557f7b8
Nov 13 13:45:33 vm1 crmd[15883]:     info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.6.1) : Non-status change
Nov 13 13:45:33 vm1 crmd[15883]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Nov 13 13:45:33 vm1 stonith-ng[15879]:     info: stonith_level_remove: Node vm3 not found (0 active entries)
Nov 13 13:45:33 vm1 stonith-ng[15879]:     info: stonith_level_register: Node vm3 has 1 active fencing levels
Nov 13 13:45:33 vm1 stonith-ng[15879]:     info: update_cib_stonith_devices: Updating device list from the cib: new resource
Nov 13 13:45:33 vm1 stonith-ng[15879]:  warning: handle_startup_fencing: Blind faith: not fencing unseen nodes
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_replace_notify: Replaced: 0.5.8 -> 0.6.1 from vm1
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Nov 13 13:45:33 vm1 crmd[15883]:     info: update_dc: Unset DC. Was vm1
Nov 13 13:45:33 vm1 attrd[15881]:   notice: attrd_cib_replaced_cb: Updating all attributes after cib_refresh_notify event
Nov 13 13:45:33 vm1 stonith-ng[15879]:     info: cib_device_update: Device F1 is allowed on vm1: score=100
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: Diff: --- 0.5.8
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: Diff: +++ 0.6.1 e65af88559035840dce69eaec2069fba
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: --         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-2383f6c"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: --         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="no-quorum-policy" value="freeze" id="cib-bootstrap-options-no-quorum-policy"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="stonith-enabled" value="true" id="cib-bootstrap-options-stonith-enabled"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="startup-fencing" value="false" id="cib-bootstrap-options-startup-fencing"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="stonith-timeout" value="60s" id="cib-bootstrap-options-stonith-timeout"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="crmd-transition-delay" value="2s" id="cib-bootstrap-options-crmd-transition-delay"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <primitive id="F1" class="stonith" type="external/libvirt">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <instance_attributes id="F1-instance_attributes">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <nvpair name="hostlist" value="vm3" id="F1-instance_attributes-hostlist"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <nvpair name="hypervisor_uri" value="qemu+ssh://bl460g1n6/system" id="F1-instance_attributes-hypervisor_uri"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </instance_attributes>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <operations>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <op name="start" interval="0s" timeout="60s" id="F1-start-0s"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <op name="monitor" interval="3600s" timeout="60s" id="F1-monitor-3600s"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <op name="stop" interval="0s" timeout="60s" id="F1-stop-0s"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </operations>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       </primitive>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <primitive id="pDummy" class="ocf" provider="pacemaker" type="Dummy">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <operations>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <op name="monitor" interval="10s" timeout="300s" on-fail="fence" id="pDummy-monitor-10s"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </operations>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       </primitive>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <rsc_location id="l1" rsc="pDummy">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <rule score="100" id="l1-rule">
Nov 13 13:45:33 vm1 crmd[15883]:     info: election_timer_cb: Election election-0 complete
Nov 13 13:45:33 vm1 crmd[15883]:     info: election_timeout_popped: Election failed: Declaring ourselves the winner
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_log: FSA: Input I_ELECTION_DC from election_timeout_popped() received in state S_ELECTION
Nov 13 13:45:33 vm1 crmd[15883]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped ]
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_takeover: Taking over DC status for this partition
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <expression attribute="#uname" operation="eq" value="vm3" id="l1-expression"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </rule>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       </rsc_location>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <rsc_location id="l2" rsc="F1">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <rule score="100" id="l2-rule">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <expression attribute="#uname" operation="eq" value="vm1" id="l2-expression"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </rule>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <rule score="100" id="l2-rule-0">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <expression attribute="#uname" operation="eq" value="vm2" id="l2-expression-0"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </rule>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <rule score="-INFINITY" id="l2-rule-1">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++           <expression attribute="#uname" operation="eq" value="vm3" id="l2-expression-1"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         </rule>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       </rsc_location>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++     <fencing-topology>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <fencing-level target="vm3" devices="F1" index="1" id="fencing"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++     </fencing-topology>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++     <rsc_defaults>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       <meta_attributes id="rsc-options">
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="resource-stickiness" value="INFINITY" id="rsc-options-resource-stickiness"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair name="migration-threshold" value="1" id="rsc-options-migration-threshold"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++       </meta_attributes>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++     </rsc_defaults>
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.6.1)
Nov 13 13:45:33 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action metadata for agent fence_legacy (target=(null))
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/32, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/33, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/34, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/35, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/36, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_destroy: Destroying 0 events
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/37, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: No such device or address (rc=-6, origin=local/crmd/38, version=0.6.1)
Nov 13 13:45:33 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.7.1
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="6" num_updates="1"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-2383f6c"/>
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/39, version=0.7.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: No such device or address (rc=-6, origin=local/crmd/40, version=0.7.1)
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm3[3232261519] - join-3 phase 4 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm1[3232261517] - join-3 phase 4 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm2[3232261518] - join-3 phase 4 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-3: Sending offer to vm3
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm3[3232261519] - join-3 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-3: Sending offer to vm1
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm1[3232261517] - join-3 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-3: Sending offer to vm2
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm2[3232261518] - join-3 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_offer_all: join-3: Waiting on 3 outstanding join acks
Nov 13 13:45:33 vm1 crmd[15883]:  warning: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm3[3232261519] - join-4 phase 1 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm1[3232261517] - join-4 phase 1 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: initialize_join: Node vm2[3232261518] - join-4 phase 1 -> 0
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-4: Sending offer to vm3
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm3[3232261519] - join-4 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-4: Sending offer to vm1
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm1[3232261517] - join-4 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: join_make_offer: join-4: Sending offer to vm2
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: join_make_offer: Node vm2[3232261518] - join-4 phase 0 -> 1
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_offer_all: join-4: Waiting on 3 outstanding join acks
Nov 13 13:45:33 vm1 crmd[15883]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:45:33 vm1 cib[15878]:   notice: log_cib_diff: cib:diff: Local-only Change: 0.8.1
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: -- <cib admin_epoch="0" epoch="7" num_updates="1"/>
Nov 13 13:45:33 vm1 cib[15878]:   notice: cib:diff: ++         <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/41, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/42, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/43, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/44, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/45, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/46, version=0.8.1)
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm1[3232261517] - join-4 phase 1 -> 2
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm3[3232261519] - join-4 phase 1 -> 2
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_filter_offer: Node vm2[3232261518] - join-4 phase 1 -> 2
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Nov 13 13:45:33 vm1 crmd[15883]:     info: crmd_join_phase_log: join-4: vm3=integrated
Nov 13 13:45:33 vm1 crmd[15883]:     info: crmd_join_phase_log: join-4: vm1=integrated
Nov 13 13:45:33 vm1 crmd[15883]:     info: crmd_join_phase_log: join-4: vm2=integrated
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_finalize: join-4: Syncing our CIB to the rest of the cluster
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/47, version=0.8.1)
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm3[3232261519] - join-4 phase 2 -> 3
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm1[3232261517] - join-4 phase 2 -> 3
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: finalize_join_for: Node vm2[3232261518] - join-4 phase 2 -> 3
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm3[3232261519] - join-4 phase 3 -> 4
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_ack: join-4: Updating node state to member for vm3
Nov 13 13:45:33 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm3']/lrm
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm1[3232261517] - join-4 phase 3 -> 4
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_ack: join-4: Updating node state to member for vm1
Nov 13 13:45:33 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm1']/lrm
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/48, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/49, version=0.8.1)
Nov 13 13:45:33 vm1 crmd[15883]:     info: crm_update_peer_join: do_dc_join_ack: Node vm2[3232261518] - join-4 phase 3 -> 4
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_dc_join_ack: join-4: Updating node state to member for vm2
Nov 13 13:45:33 vm1 crmd[15883]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm2']/lrm
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/50, version=0.8.1)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm3']/lrm: OK (rc=0, origin=local/crmd/51, version=0.8.2)
Nov 13 13:45:33 vm1 cib[15933]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/52, version=0.8.3)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm1']/lrm: OK (rc=0, origin=local/crmd/53, version=0.8.4)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/54, version=0.8.5)
Nov 13 13:45:33 vm1 cib[15933]:     info: write_cib_contents: Wrote version 0.8.0 of the CIB to disk (digest: 9db35554f5ac4e48336f1bae33d89abc)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='vm2']/lrm: OK (rc=0, origin=local/crmd/55, version=0.8.6)
Nov 13 13:45:33 vm1 cib[15933]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.aXeGdJ (digest: /var/lib/pacemaker/cib/cib.0mEUho)
Nov 13 13:45:33 vm1 crmd[15883]:     info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Nov 13 13:45:33 vm1 crmd[15883]:     info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_new: Connecting 0x156e4b0 for uid=0 gid=0 pid=15932 id=e6fecbad-5798-4b7d-bb03-c2062e3f6462
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/56, version=0.8.7)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/57, version=0.8.7)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/58, version=0.8.7)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/59, version=0.8.7)
Nov 13 13:45:33 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.8.7)
Nov 13 13:45:33 vm1 cib[15878]:     info: crm_client_destroy: Destroying 0 events
Nov 13 13:45:34 vm1 stonith-ng[15879]:   notice: stonith_device_register: Added 'F1' to the device list (1 active devices)
Nov 13 13:45:35 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:45:35 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/60, version=0.8.7)
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:45:35 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:45:35 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Stopped 
Nov 13 13:45:35 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	Stopped 
Nov 13 13:45:35 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (3600s) for F1 on vm1
Nov 13 13:45:35 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm3
Nov 13 13:45:35 vm1 pengine[15882]:   notice: LogActions: Start   F1	(vm1)
Nov 13 13:45:35 vm1 pengine[15882]:   notice: LogActions: Start   pDummy	(vm3)
Nov 13 13:45:35 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:45:35 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1384317935-33) derived from /var/lib/pacemaker/pengine/pe-input-1.bz2
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 10: monitor F1_monitor_0 on vm3
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 7: monitor F1_monitor_0 on vm2
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 4: monitor F1_monitor_0 on vm1 (local)
Nov 13 13:45:35 vm1 lrmd[15880]:     info: process_lrmd_get_rsc_info: Resource 'F1' not found (0 active resources)
Nov 13 13:45:35 vm1 lrmd[15880]:     info: process_lrmd_rsc_register: Added 'F1' to the rsc list (1 active resources)
Nov 13 13:45:35 vm1 crmd[15883]:     info: do_lrm_rsc_op: Performing key=4:1:7:154fb289-24e8-407e-9a03-69a510480b60 op=F1_monitor_0
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 11: monitor pDummy_monitor_0 on vm3
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 8: monitor pDummy_monitor_0 on vm2
Nov 13 13:45:35 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 5: monitor pDummy_monitor_0 on vm1 (local)
Nov 13 13:45:35 vm1 pengine[15882]:   notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-1.bz2
Nov 13 13:45:35 vm1 stonith-ng[15879]:     info: crm_client_new: Connecting 0x1994cd0 for uid=0 gid=0 pid=15880 id=e2d2e964-db5b-4c6f-925f-8d0c9a6e6299
Nov 13 13:45:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed register from lrmd.15880: OK (0)
Nov 13 13:45:35 vm1 lrmd[15880]:     info: process_lrmd_get_rsc_info: Resource 'pDummy' not found (1 active resources)
Nov 13 13:45:35 vm1 lrmd[15880]:     info: process_lrmd_rsc_register: Added 'pDummy' to the rsc list (2 active resources)
Nov 13 13:45:35 vm1 crmd[15883]:     info: do_lrm_rsc_op: Performing key=5:1:7:154fb289-24e8-407e-9a03-69a510480b60 op=pDummy_monitor_0
Nov 13 13:45:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify from lrmd.15880: OK (0)
Nov 13 13:45:35 vm1 Dummy(pDummy)[15934]: DEBUG: pDummy monitor : 7
Nov 13 13:45:36 vm1 crmd[15883]:     info: process_lrm_event: LRM operation F1_monitor_0 (call=5, rc=7, cib-update=61, confirmed=true) not running
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/crmd/17, version=0.8.8)
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/61, version=0.8.9)
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm3/crmd/17, version=0.8.10)
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/crmd/18, version=0.8.11)
Nov 13 13:45:36 vm1 crmd[15883]:     info: services_os_action_execute: Managed Dummy_meta-data_0 process 15955 exited with rc=0
Nov 13 13:45:36 vm1 crmd[15883]:   notice: process_lrm_event: LRM operation pDummy_monitor_0 (call=9, rc=7, cib-update=62, confirmed=true) not running
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/62, version=0.8.12)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action F1_monitor_0 (7) confirmed on vm2 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action F1_monitor_0 (4) confirmed on vm1 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action F1_monitor_0 (10) confirmed on vm3 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action pDummy_monitor_0 (8) confirmed on vm2 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action pDummy_monitor_0 (5) confirmed on vm1 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 6: probe_complete probe_complete on vm2 - no waiting
Nov 13 13:45:36 vm1 crmd[15883]:     info: te_rsc_command: Action 6 confirmed - no wait
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 3: probe_complete probe_complete on vm1 (local) - no waiting
Nov 13 13:45:36 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting probe_complete[vm1] = true
Nov 13 13:45:36 vm1 crmd[15883]:     info: te_rsc_command: Action 3 confirmed - no wait
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action pDummy_monitor_0 (11) confirmed on vm3 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 9: probe_complete probe_complete on vm3 - no waiting
Nov 13 13:45:36 vm1 crmd[15883]:     info: te_rsc_command: Action 9 confirmed - no wait
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 12: start F1_start_0 on vm1 (local)
Nov 13 13:45:36 vm1 crmd[15883]:     info: do_lrm_rsc_op: Performing key=12:1:0:154fb289-24e8-407e-9a03-69a510480b60 op=F1_start_0
Nov 13 13:45:36 vm1 lrmd[15880]:     info: log_execute: executing - rsc:F1 action:start call_id:10
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 14: start pDummy_start_0 on vm3
Nov 13 13:45:36 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action metadata for agent fence_legacy (target=(null))
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm3/crmd/18, version=0.8.13)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action pDummy_start_0 (14) confirmed on vm3 (rc=0)
Nov 13 13:45:36 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 15: monitor pDummy_monitor_10000 on vm3
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm3/crmd/19, version=0.8.14)
Nov 13 13:45:36 vm1 crmd[15883]:     info: match_graph_event: Action pDummy_monitor_10000 (15) confirmed on vm3 (rc=0)
Nov 13 13:45:36 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm3/crmd/20, version=0.8.15)
Nov 13 13:45:37 vm1 stonith-ng[15879]:   notice: stonith_device_register: Device 'F1' already existed in device list (1 active devices)
Nov 13 13:45:37 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_device_register from lrmd.15880: OK (0)
Nov 13 13:45:37 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_execute from lrmd.15880: Operation now in progress (-115)
Nov 13 13:45:37 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action monitor for agent fence_legacy (target=(null))
Nov 13 13:45:38 vm1 stonith-ng[15879]:     info: log_operation: F1:15985 [ Performing: stonith -t external/libvirt -S ]
Nov 13 13:45:38 vm1 stonith-ng[15879]:     info: log_operation: F1:15985 [ success:  0 ]
Nov 13 13:45:38 vm1 lrmd[15880]:     info: log_finished: finished - rsc:F1 action:start call_id:10  exit-code:0 exec-time:2305ms queue-time:0ms
Nov 13 13:45:38 vm1 crmd[15883]:   notice: process_lrm_event: LRM operation F1_start_0 (call=10, rc=0, cib-update=63, confirmed=true) ok
Nov 13 13:45:38 vm1 crmd[15883]:     info: match_graph_event: Action F1_start_0 (12) confirmed on vm1 (rc=0)
Nov 13 13:45:38 vm1 crmd[15883]:   notice: te_rsc_command: Initiating action 13: monitor F1_monitor_3600000 on vm1 (local)
Nov 13 13:45:38 vm1 crmd[15883]:     info: do_lrm_rsc_op: Performing key=13:1:0:154fb289-24e8-407e-9a03-69a510480b60 op=F1_monitor_3600000
Nov 13 13:45:38 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/63, version=0.8.16)
Nov 13 13:45:38 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_execute from lrmd.15880: Operation now in progress (-115)
Nov 13 13:45:38 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action monitor for agent fence_legacy (target=(null))
Nov 13 13:45:39 vm1 stonith-ng[15879]:     info: log_operation: F1:16001 [ Performing: stonith -t external/libvirt -S ]
Nov 13 13:45:39 vm1 stonith-ng[15879]:     info: log_operation: F1:16001 [ success:  0 ]
Nov 13 13:45:39 vm1 crmd[15883]:   notice: process_lrm_event: LRM operation F1_monitor_3600000 (call=11, rc=0, cib-update=64, confirmed=false) ok
Nov 13 13:45:39 vm1 crmd[15883]:     info: match_graph_event: Action F1_monitor_3600000 (13) confirmed on vm1 (rc=0)
Nov 13 13:45:39 vm1 crmd[15883]:   notice: run_graph: Transition 1 (Complete=14, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1.bz2): Complete
Nov 13 13:45:39 vm1 crmd[15883]:     info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Nov 13 13:45:39 vm1 crmd[15883]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 13 13:45:39 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/64, version=0.8.17)
Nov 13 13:47:06 vm1 crmd[15883]:     info: abort_transition_graph: process_graph_event:583 - Triggered transition abort (complete=1, node=vm3, tag=lrm_rsc_op, id=pDummy_last_failure_0, magic=0:7;15:1:0:154fb289-24e8-407e-9a03-69a510480b60, cib=0.8.18) : Inactive graph
Nov 13 13:47:06 vm1 crmd[15883]:  warning: update_failcount: Updating failcount for pDummy on vm3 after failed monitor: rc=7 (update=value++, time=1384318026)
Nov 13 13:47:06 vm1 attrd[15881]:     info: attrd_client_message: Expanded fail-count-pDummy=value++ to 1
Nov 13 13:47:06 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting fail-count-pDummy[vm3] = 1
Nov 13 13:47:06 vm1 crmd[15883]:     info: process_graph_event: Detected action (1.15) pDummy_monitor_10000.11=not running: failed
Nov 13 13:47:06 vm1 attrd[15881]:     info: attrd_client_message: Broadcasting last-failure-pDummy[vm3] = 1384318026
Nov 13 13:47:06 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm3/crmd/21, version=0.8.18)
Nov 13 13:47:06 vm1 crmd[15883]:     info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=1, node=vm3, tag=nvpair, id=status-3232261519-fail-count-pDummy, name=fail-count-pDummy, value=1, magic=NA, cib=0.8.19) : Transient attribute: update
Nov 13 13:47:06 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/6, version=0.8.19)
Nov 13 13:47:06 vm1 crmd[15883]:     info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=1, node=vm3, tag=nvpair, id=status-3232261519-last-failure-pDummy, name=last-failure-pDummy, value=1384318026, magic=NA, cib=0.8.20) : Transient attribute: update
Nov 13 13:47:06 vm1 cib[15878]:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vm2/attrd/7, version=0.8.20)
Nov 13 13:47:08 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:08 vm1 crmd[15883]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:08 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:08 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/65, version=0.8.20)
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:08 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:08 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:08 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:08 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:08 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:08 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:08 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:08 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:08 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:08 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:08 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:08 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:08 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:08 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1384318028-47) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:08 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:08 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:08 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 696fb2c3-e11a-4124-ba9b-bafc9ab28426 (0)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action list for agent fence_legacy (target=(null))
Nov 13 13:47:08 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 2 of 3 from vm2 (1 devices)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.696fb2c3
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: dynamic_list_search_cb: Refreshing port list for F1
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:08 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:12 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:12 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.696fb2c3 failed
Nov 13 13:47:12 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:12 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.696fb2c3: Generic Pacemaker error
Nov 13 13:47:12 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 2/13:2:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:12 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 2 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:12 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:12 vm1 crmd[15883]:   notice: run_graph: Transition 2 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:12 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:12 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=696fb2c3-e11a-4124-ba9b-bafc9ab28426) by client crmd.15883
Nov 13 13:47:14 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:14 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:14 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:14 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/66, version=0.8.20)
Nov 13 13:47:14 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:14 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:14 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:14 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:14 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:14 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:14 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:14 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:14 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:14 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:14 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:14 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:14 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:14 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:14 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1384318034-48) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:14 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:14 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:14 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 431c7488-013e-4900-bde7-a3ce154b35a3 (0)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:14 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:14 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.431c7488
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:14 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:17 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:17 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.431c7488 failed
Nov 13 13:47:17 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:17 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.431c7488: Generic Pacemaker error
Nov 13 13:47:17 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:17 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 3/13:3:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:17 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 3 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:17 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:17 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=431c7488-013e-4900-bde7-a3ce154b35a3) by client crmd.15883
Nov 13 13:47:17 vm1 crmd[15883]:   notice: run_graph: Transition 3 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:19 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:19 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:19 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:19 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/67, version=0.8.20)
Nov 13 13:47:19 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:19 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:19 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:19 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:19 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:19 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:19 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:19 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:19 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:19 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:19 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:19 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:19 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:19 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:19 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1384318039-49) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:19 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:19 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:19 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 682bdc12-35a4-431a-8773-4862cc8c39ef (0)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:19 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:19 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.682bdc12
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:19 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:22 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:22 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.682bdc12 failed
Nov 13 13:47:22 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:22 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.682bdc12: Generic Pacemaker error
Nov 13 13:47:22 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:22 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 4/13:4:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:22 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 4 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:22 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:22 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=682bdc12-35a4-431a-8773-4862cc8c39ef) by client crmd.15883
Nov 13 13:47:22 vm1 crmd[15883]:   notice: run_graph: Transition 4 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:24 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:24 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:24 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:24 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/68, version=0.8.20)
Nov 13 13:47:24 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:24 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:24 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:24 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:24 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:24 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:24 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:24 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:24 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:24 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:24 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:24 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:24 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:24 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:24 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1384318044-50) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:24 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:24 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:24 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: d761e73f-f337-48cc-b2a1-5b2d722d2738 (0)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:24 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 5: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:24 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.d761e73f
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:24 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:27 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:27 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.d761e73f failed
Nov 13 13:47:27 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:27 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.d761e73f: Generic Pacemaker error
Nov 13 13:47:27 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:27 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 5/13:5:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:27 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 5 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:27 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:27 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=d761e73f-f337-48cc-b2a1-5b2d722d2738) by client crmd.15883
Nov 13 13:47:27 vm1 crmd[15883]:   notice: run_graph: Transition 5 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:29 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:29 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:29 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:29 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/69, version=0.8.20)
Nov 13 13:47:29 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:29 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:29 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:29 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:29 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:29 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:29 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:29 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:29 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:29 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:29 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:29 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:29 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:29 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:29 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1384318049-51) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:29 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:29 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:29 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 11df91ab-fc81-43aa-941d-ffa1204df1c9 (0)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:29 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:29 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 2 of 3 from vm1 (1 devices)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.11df91ab
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm1 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm2 (1 devices)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:29 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:30 vm1 stonith: [16072]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:30 vm1 stonith-ng[15879]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:33 vm1 stonith: [16084]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:33 vm1 stonith-ng[15879]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:33 vm1 stonith-ng[15879]:    error: log_operation: Operation 'reboot' [16083] (call 6 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:33 vm1 stonith-ng[15879]:  warning: log_operation: F1:16083 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:33 vm1 stonith-ng[15879]:  warning: log_operation: F1:16083 [ failed: vm3 5 ]
Nov 13 13:47:33 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:33 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.11df91ab failed
Nov 13 13:47:33 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm1: OK (0)
Nov 13 13:47:33 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.11df91ab: Generic Pacemaker error
Nov 13 13:47:33 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 6/13:6:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:33 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 6 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:33 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:33 vm1 crmd[15883]:   notice: run_graph: Transition 6 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:33 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=11df91ab-fc81-43aa-941d-ffa1204df1c9) by client crmd.15883
Nov 13 13:47:33 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:35 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:35 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:35 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:35 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/70, version=0.8.20)
Nov 13 13:47:35 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:35 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:35 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:35 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:35 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:35 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:35 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:35 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:35 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:35 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:35 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:35 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:35 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:35 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:35 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1384318055-52) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:35 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:35 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:35 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 84777767-aa8b-4e04-8dec-b26dae36aaff (0)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:35 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 7: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:35 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.84777767
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:35 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:38 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:38 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.84777767 failed
Nov 13 13:47:38 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:38 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.84777767: Generic Pacemaker error
Nov 13 13:47:38 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:38 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 7/13:7:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:38 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 7 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:38 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:38 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=84777767-aa8b-4e04-8dec-b26dae36aaff) by client crmd.15883
Nov 13 13:47:38 vm1 crmd[15883]:   notice: run_graph: Transition 7 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:40 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:40 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:40 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:40 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/71, version=0.8.20)
Nov 13 13:47:40 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:40 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:40 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:40 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:40 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:40 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:40 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:40 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:40 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:40 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:40 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:40 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:40 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:40 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:40 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1384318060-53) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:40 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:40 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:40 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 588ca7d3-cb01-4e1b-9c7d-5fcdd5b66a27 (0)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:40 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 8: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:40 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.588ca7d3
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:40 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:43 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:43 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.588ca7d3 failed
Nov 13 13:47:43 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:43 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.588ca7d3: Generic Pacemaker error
Nov 13 13:47:43 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:43 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 8/13:8:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:43 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 8 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:43 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:43 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=588ca7d3-cb01-4e1b-9c7d-5fcdd5b66a27) by client crmd.15883
Nov 13 13:47:43 vm1 crmd[15883]:   notice: run_graph: Transition 8 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:45 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:45 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:45 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:45 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/72, version=0.8.20)
Nov 13 13:47:45 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:45 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:45 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:45 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:45 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:45 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:45 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:45 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:45 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:45 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:45 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:45 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:45 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:45 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:45 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1384318065-54) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:45 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:45 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:45 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: a3379e0c-d206-4ced-9e7e-1c915f08a0ae (0)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:45 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 9: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:45 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.a3379e0c
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:45 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:48 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:48 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.a3379e0c failed
Nov 13 13:47:48 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:48 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.a3379e0c: Generic Pacemaker error
Nov 13 13:47:48 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 9/13:9:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:48 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 9 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:48 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:48 vm1 crmd[15883]:   notice: run_graph: Transition 9 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:48 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:48 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=a3379e0c-d206-4ced-9e7e-1c915f08a0ae) by client crmd.15883
Nov 13 13:47:50 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:50 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:50 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:50 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/73, version=0.8.20)
Nov 13 13:47:50 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:50 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:50 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:50 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:50 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:50 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:50 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:50 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:50 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:50 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:50 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:50 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:50 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:50 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:50 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1384318070-55) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:50 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:50 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:50 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 9ab4c26b-da3e-40cd-ba98-c89017db4953 (0)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:50 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 10: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:50 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.9ab4c26b
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:50 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:54 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:54 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.9ab4c26b failed
Nov 13 13:47:54 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:54 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.9ab4c26b: Generic Pacemaker error
Nov 13 13:47:54 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:54 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 10/13:10:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:54 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 10 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:54 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:54 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=9ab4c26b-da3e-40cd-ba98-c89017db4953) by client crmd.15883
Nov 13 13:47:54 vm1 crmd[15883]:   notice: run_graph: Transition 10 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:47:56 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:47:56 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:47:56 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:47:56 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/74, version=0.8.20)
Nov 13 13:47:56 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:47:56 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:47:56 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:47:56 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:47:56 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:47:56 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:47:56 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:47:56 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:47:56 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:47:56 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:47:56 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:47:56 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:47:56 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:47:56 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:47:56 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 11 (ref=pe_calc-dc-1384318076-56) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:56 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:47:56 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:47:56 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 1ba836f2-328d-45c7-adbb-1db9b0a1ca4c (0)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:47:56 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 11: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:47:56 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.1ba836f2
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:47:56 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:47:59 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:47:59 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.1ba836f2 failed
Nov 13 13:47:59 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:47:59 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.1ba836f2: Generic Pacemaker error
Nov 13 13:47:59 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:59 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 11/13:11:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:47:59 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 11 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:47:59 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:47:59 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=1ba836f2-328d-45c7-adbb-1db9b0a1ca4c) by client crmd.15883
Nov 13 13:47:59 vm1 crmd[15883]:   notice: run_graph: Transition 11 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:48:01 vm1 crmd[15883]:     info: crm_timer_popped: New Transition Timer (I_PE_CALC) just popped (2000ms)
Nov 13 13:48:01 vm1 crmd[15883]:     info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 13:48:01 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 13:48:01 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/75, version=0.8.20)
Nov 13 13:48:01 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 13:48:01 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 13:48:01 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 13:48:01 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 13:48:01 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 13:48:01 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 13:48:01 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 13:48:01 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 13:48:01 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 13:48:01 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 13:48:01 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 13:48:01 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 13:48:01 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 13:48:01 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 13:48:01 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 12 (ref=pe_calc-dc-1384318081-57) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:48:01 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 13:48:01 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 13:48:01 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 00825b71-24e3-4f14-a0b8-6945f050dfd1 (0)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 13:48:01 vm1 stonith-ng[15879]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 1 of 3 from vm2 (1 devices)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.00825b71
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm2 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 13:48:01 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 12: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm1 (1 devices)
Nov 13 13:48:01 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 13:48:04 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 13:48:04 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.00825b71 failed
Nov 13 13:48:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm2: OK (0)
Nov 13 13:48:04 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm2 for crmd.15883@vm1.00825b71: Generic Pacemaker error
Nov 13 13:48:04 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 12/13:12:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 13:48:04 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 12 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 13:48:04 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 13:48:04 vm1 crmd[15883]:   notice: run_graph: Transition 12 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 13:48:04 vm1 crmd[15883]:   notice: too_many_st_failures: Too many failures to fence vm3 (11), giving up
Nov 13 13:48:04 vm1 crmd[15883]:     info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Nov 13 13:48:04 vm1 crmd[15883]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 13 13:48:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:48:04 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm2 for vm1: Generic Pacemaker error (ref=00825b71-24e3-4f14-a0b8-6945f050dfd1) by client crmd.15883
Nov 13 14:03:04 vm1 crmd[15883]:     info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
Nov 13 14:03:04 vm1 crmd[15883]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Nov 13 14:03:04 vm1 crmd[15883]:     info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Nov 13 14:03:04 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/76, version=0.8.20)
Nov 13 14:03:04 vm1 pengine[15882]:     info: process_pe_message: Input has not changed since last time, not saving to disk
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm3 is active
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status: Node vm3 is online
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm1 is active
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status: Node vm1 is online
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status_fencing: Node vm2 is active
Nov 13 14:03:04 vm1 pengine[15882]:     info: determine_online_status: Node vm2 is online
Nov 13 14:03:04 vm1 pengine[15882]:  warning: unpack_rsc_op_failure: Processing failed op monitor for pDummy on vm3: not running (7)
Nov 13 14:03:04 vm1 pengine[15882]:  warning: pe_fence_node: Node vm3 will be fenced because of resource failure(s)
Nov 13 14:03:04 vm1 pengine[15882]:     info: native_print: F1	(stonith:external/libvirt):	Started vm1 
Nov 13 14:03:04 vm1 pengine[15882]:     info: native_print: pDummy	(ocf::pacemaker:Dummy):	FAILED vm3 
Nov 13 14:03:04 vm1 pengine[15882]:     info: get_failcount_full: pDummy has failed 1 times on vm3
Nov 13 14:03:04 vm1 pengine[15882]:  warning: common_apply_stickiness: Forcing pDummy away from vm3 after 1 failures (max=1)
Nov 13 14:03:04 vm1 pengine[15882]:     info: RecurringOp:  Start recurring monitor (10s) for pDummy on vm2
Nov 13 14:03:04 vm1 pengine[15882]:  warning: stage6: Scheduling Node vm3 for STONITH
Nov 13 14:03:04 vm1 pengine[15882]:   notice: native_stop_constraints: Stop of failed resource pDummy is implicit after vm3 is fenced
Nov 13 14:03:04 vm1 pengine[15882]:     info: LogActions: Leave   F1	(Started vm1)
Nov 13 14:03:04 vm1 pengine[15882]:   notice: LogActions: Recover pDummy	(Started vm3 -> vm2)
Nov 13 14:03:04 vm1 crmd[15883]:     info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Nov 13 14:03:04 vm1 crmd[15883]:     info: do_te_invoke: Processing graph 13 (ref=pe_calc-dc-1384318984-58) derived from /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 14:03:04 vm1 crmd[15883]:   notice: te_fence_node: Executing reboot fencing operation (13) on vm3 (timeout=60000)
Nov 13 14:03:04 vm1 stonith-ng[15879]:   notice: handle_request: Client crmd.15883.e6cedc3f wants to fence (reboot) 'vm3' with device '(any)'
Nov 13 14:03:04 vm1 stonith-ng[15879]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for vm3: 893bcd8c-11ea-4f2c-b5d5-e2c9d3883c1b (0)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from crmd.15883: Operation now in progress (-115)
Nov 13 14:03:04 vm1 pengine[15882]:  warning: process_pe_message: Calculated Transition 13: /var/lib/pacemaker/pengine/pe-warn-0.bz2
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action list for agent fence_legacy (target=(null))
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm3: OK (0)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: dynamic_list_search_cb: Refreshing port list for F1
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 2 of 3 from vm1 (1 devices)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node vm3 for crmd.15883.893bcd8c
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: call_remote_stonith: Requesting that vm1 perform op reboot vm3 with F1 for crmd.15883 (72s)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm1: OK (0)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: process_remote_stonith_query: Query result 3 of 3 from vm2 (1 devices)
Nov 13 14:03:04 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_query reply from vm2: OK (0)
Nov 13 14:03:05 vm1 stonith: [16337]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 14:03:05 vm1 stonith-ng[15879]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 14:03:07 vm1 root: Mark:pcmk:1384318987
Nov 13 14:03:07 vm1 stonith: [16391]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 14:03:07 vm1 stonith-ng[15879]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 14:03:07 vm1 stonith-ng[15879]:    error: log_operation: Operation 'reboot' [16348] (call 13 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 14:03:07 vm1 stonith-ng[15879]:  warning: log_operation: F1:16348 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 14:03:07 vm1 stonith-ng[15879]:  warning: log_operation: F1:16348 [ failed: vm3 5 ]
Nov 13 14:03:07 vm1 stonith-ng[15879]:   notice: process_remote_stonith_exec: Call to F1 for vm3 on behalf of crmd.15883@vm1: Generic Pacemaker error (-201)
Nov 13 14:03:07 vm1 stonith-ng[15879]:   notice: stonith_topology_next: All fencing options to fence vm3 for crmd.15883@vm1.893bcd8c failed
Nov 13 14:03:07 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_fence reply from vm1: OK (0)
Nov 13 14:03:07 vm1 stonith-ng[15879]:    error: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.893bcd8c: Generic Pacemaker error
Nov 13 14:03:07 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 13/13:13:0:154fb289-24e8-407e-9a03-69a510480b60: Generic Pacemaker error (-201)
Nov 13 14:03:07 vm1 crmd[15883]:   notice: tengine_stonith_callback: Stonith operation 13 for vm3 failed (Generic Pacemaker error): aborting transition.
Nov 13 14:03:07 vm1 crmd[15883]:     info: abort_transition_graph: tengine_stonith_callback:463 - Triggered transition abort (complete=0) : Stonith failed
Nov 13 14:03:07 vm1 crmd[15883]:   notice: run_graph: Transition 13 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
Nov 13 14:03:07 vm1 crmd[15883]:   notice: too_many_st_failures: Too many failures to fence vm3 (12), giving up
Nov 13 14:03:07 vm1 crmd[15883]:     info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Nov 13 14:03:07 vm1 crmd[15883]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 13 14:03:07 vm1 crmd[15883]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=893bcd8c-11ea-4f2c-b5d5-e2c9d3883c1b) by client crmd.15883
Nov 13 14:03:07 vm1 stonith-ng[15879]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 14:03:16 vm1 cib[15878]:     info: crm_client_new: Connecting 0x16104a0 for uid=0 gid=0 pid=17477 id=41ad2846-b251-4248-ba10-3fc2b7a69936
Nov 13 14:03:16 vm1 cib[15878]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.8.20)
Nov 13 14:03:16 vm1 cib[15878]:     info: crm_client_destroy: Destroying 0 events
Nov 13 14:03:16 vm1 cib[15878]:     info: crm_client_new: Connecting 0x16104a0 for uid=0 gid=0 pid=17479 id=83c69c76-cf00-4b39-b471-d2ed0145a0f3
