Nov 13 13:44:17 vm2 corosync[31444]:   [MAIN  ] main.c:main:1171 Corosync Cluster Engine ('2.3.2.7-a911'): started and ready to provide service.
Nov 13 13:44:17 vm2 corosync[31444]:   [MAIN  ] main.c:main:1172 Corosync built-in features: watchdog upstart snmp pie relro bindnow
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemnet.c:totemnet_instance_initialize:242 Initializing transport (UDP/IP Multicast).
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemcrypto.c:init_nss:579 Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemnet.c:totemnet_instance_initialize:242 Initializing transport (UDP/IP Multicast).
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemcrypto.c:init_nss:579 Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemudp.c:timer_function_netif_check_timeout:670 The network interface [192.168.101.142] is now up.
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync configuration map access [0]
Nov 13 13:44:17 vm2 corosync[31445]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cmap
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync configuration service [1]
Nov 13 13:44:17 vm2 corosync[31445]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cfg
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync cluster closed process group service v1.01 [2]
Nov 13 13:44:17 vm2 corosync[31445]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: cpg
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync profile loading service [4]
Nov 13 13:44:17 vm2 corosync[31445]:   [WD    ] wd.c:setup_watchdog:631 No Watchdog, try modprobe <a watchdog>
Nov 13 13:44:17 vm2 corosync[31445]:   [WD    ] wd.c:wd_scan_resources:580 no resources configured.
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync watchdog service [7]
Nov 13 13:44:17 vm2 corosync[31445]:   [QUORUM] vsf_quorum.c:quorum_exec_init_fn:274 Using quorum provider corosync_votequorum
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync vote quorum service v1.0 [5]
Nov 13 13:44:17 vm2 corosync[31445]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: votequorum
Nov 13 13:44:17 vm2 corosync[31445]:   [SERV  ] service.c:corosync_service_link_and_init:174 Service engine loaded: corosync cluster quorum service v0.1 [3]
Nov 13 13:44:17 vm2 corosync[31445]:   [QB    ] ipc_setup.c:qb_ipcs_us_publish:377 server name: quorum
Nov 13 13:44:17 vm2 corosync[31445]:   [TOTEM ] totemudp.c:timer_function_netif_check_timeout:670 The network interface [192.168.102.142] is now up.
Nov 13 13:44:18 vm2 corosync[31445]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.142:4) was formed. Members joined: -1062705778
Nov 13 13:44:18 vm2 corosync[31445]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[1]: -1062705778
Nov 13 13:44:18 vm2 corosync[31445]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:18 vm2 corosync[31445]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.141:8) was formed. Members joined: -1062705779
Nov 13 13:44:18 vm2 corosync[31445]:   [QUORUM] vsf_quorum.c:quorum_api_set_quorum:148 This node is within the primary component and will provide service.
Nov 13 13:44:18 vm2 corosync[31445]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[2]: -1062705779 -1062705778
Nov 13 13:44:18 vm2 corosync[31445]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:18 vm2 corosync[31445]:   [TOTEM ] totemsrp.c:memb_state_operational_enter:2016 A new membership (192.168.101.141:12) was formed. Members joined: -1062705777
Nov 13 13:44:18 vm2 corosync[31445]:   [QUORUM] vsf_quorum.c:log_view_list:132 Members[3]: -1062705779 -1062705778 -1062705777
Nov 13 13:44:18 vm2 corosync[31445]:   [MAIN  ] main.c:corosync_sync_completed:276 Completed service synchronization, ready to provide service.
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 pacemakerd[31455]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_ipc_connect: Could not establish pacemakerd connection: Connection refused (111)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: get_cluster_type: Detected an active 'corosync' cluster
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: mcp_read_config: Reading configure for stack: corosync
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: mcp_read_config: Configured corosync to accept connections from group 492: OK (1)
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: main: Starting Pacemaker 1.1.10 (Build: 2383f6c):  ncurses libqb-logging libqb-ipc lha-fencing nagios  corosync-native snmp
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: main: Maximum core file size is: 18446744073709551615
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: qb_ipcs_us_publish: server name: pacemakerd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Created entry 3233eac7-ee3c-4168-b5a5-d00bb986e0ef/0x1d8e030 for node (null)/3232261518 (1 total)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: cluster_connect_quorum: Quorum acquired
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Using uid=496 and group=492 for process cib
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31459 for process cib
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31460 for process stonith-ng
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31461 for process lrmd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Using uid=496 and group=492 for process attrd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31462 for process attrd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Using uid=496 and group=492 for process pengine
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31463 for process pengine
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Using uid=496 and group=492 for process crmd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: start_child: Forked child 31464 for process crmd
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: main: Starting mainloop
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: pcmk_quorum_notification: Membership 12: quorum retained (3)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Created entry 1e2d1f4f-c6fd-4255-8cbe-137571cb45e8/0x1e903d0 for node (null)/3232261517 (2 total)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261517
Nov 13 13:44:20 vm2 cib[31459]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 cib[31459]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 attrd[31462]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 lrmd[31461]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 lrmd[31461]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 lrmd[31461]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm2 lrmd[31461]:     info: qb_ipcs_us_publish: server name: lrmd
Nov 13 13:44:20 vm2 attrd[31462]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm2 attrd[31462]:     info: main: Starting up
Nov 13 13:44:20 vm2 attrd[31462]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm2 stonith-ng[31460]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 stonith-ng[31460]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm2 cib[31459]:   notice: main: Using new config location: /var/lib/pacemaker/cib
Nov 13 13:44:20 vm2 cib[31459]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm2 cib[31459]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm2 lrmd[31461]:     info: main: Starting
Nov 13 13:44:20 vm2 attrd[31462]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm2 attrd[31462]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm2 cib[31459]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
Nov 13 13:44:20 vm2 cib[31459]:  warning: retrieveCib: Cluster configuration not found: /var/lib/pacemaker/cib/cib.xml
Nov 13 13:44:20 vm2 cib[31459]:  warning: readCibXmlFile: Primary configuration corrupt or unusable, trying backups in /var/lib/pacemaker/cib
Nov 13 13:44:20 vm2 cib[31459]:  warning: readCibXmlFile: Continuing with an empty configuration.
Nov 13 13:44:20 vm2 cib[31459]:     info: validate_with_relaxng: Creating RNG parser context
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm2 stonith-ng[31460]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm2 pengine[31463]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 pengine[31463]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 pengine[31463]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm2 pengine[31463]:     info: qb_ipcs_us_publish: server name: pengine
Nov 13 13:44:20 vm2 pengine[31463]:     info: main: Starting pengine
Nov 13 13:44:20 vm2 crmd[31464]:   notice: crm_add_logfile: Additional logging available in /var/log/ha-debug
Nov 13 13:44:20 vm2 crmd[31464]:    debug: crm_update_callsites: Enabling callsites based on priority=7, files=remote.c,commands.c,main.c, functions=(null), formats=(null), tags=(null)
Nov 13 13:44:20 vm2 crmd[31464]:     info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/hacluster
Nov 13 13:44:20 vm2 crmd[31464]:   notice: main: CRM Git Version: 2383f6c
Nov 13 13:44:20 vm2 crmd[31464]:     info: do_log: FSA: Input I_STARTUP from crmd_init() received in state S_STARTING
Nov 13 13:44:20 vm2 crmd[31464]:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 13 13:44:20 vm2 crmd[31464]:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 13 13:44:20 vm2 crmd[31464]:     info: crm_ipc_connect: Could not establish cib_shm connection: Connection refused (111)
Nov 13 13:44:20 vm2 cib[31459]:     info: startCib: CIB Initialization completed successfully
Nov 13 13:44:20 vm2 cib[31459]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261517] - state is now member (was (null))
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node vm2[3232261518] - state is now member (was (null))
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Created entry 191e0eaf-2935-49e5-8ffe-b876b3112438/0x1e8f7d0 for node (null)/3232261519 (3 total)
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261519
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_get_peer: Created entry 82a2ec4c-d6d9-47d4-9c8f-01d206912a35/0x871130 for node (null)/3232261518 (1 total)
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:20 vm2 attrd[31462]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261518] - state is now member (was (null))
Nov 13 13:44:20 vm2 attrd[31462]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_get_peer: Created entry 45e8e5d2-7778-4e42-95b0-f4f07035d0c8/0xfe96a0 for node (null)/3232261518 (1 total)
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_get_peer: Created entry ab16e495-b8ed-4471-b158-1e5d2ae60c1a/0xbc9360 for node (null)/3232261518 (1 total)
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:20 vm2 cib[31459]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261519
Nov 13 13:44:20 vm2 pacemakerd[31455]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:20 vm2 pacemakerd[31455]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:20 vm2 attrd[31462]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:20 vm2 attrd[31462]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:20 vm2 attrd[31462]:     info: main: Cluster connection active
Nov 13 13:44:20 vm2 attrd[31462]:     info: qb_ipcs_us_publish: server name: attrd
Nov 13 13:44:20 vm2 attrd[31462]:     info: main: Accepting attribute updates
Nov 13 13:44:20 vm2 attrd[31462]:     info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
Nov 13 13:44:20 vm2 stonith-ng[31460]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:20 vm2 stonith-ng[31460]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:20 vm2 stonith-ng[31460]:     info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
Nov 13 13:44:20 vm2 cib[31459]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:20 vm2 cib[31459]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:20 vm2 cib[31459]:     info: qb_ipcs_us_publish: server name: cib_ro
Nov 13 13:44:20 vm2 cib[31459]:     info: qb_ipcs_us_publish: server name: cib_rw
Nov 13 13:44:20 vm2 cib[31459]:     info: qb_ipcs_us_publish: server name: cib_shm
Nov 13 13:44:20 vm2 cib[31459]:     info: cib_init: Starting cib mainloop
Nov 13 13:44:20 vm2 cib[31459]:     info: pcmk_cpg_membership: Joined[0.0] cib.3232261518 
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_get_peer: Created entry b2901563-7693-4187-84ce-efa76e0af4c1/0xbcbbc0 for node (null)/3232261517 (2 total)
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:20 vm2 cib[31459]:     info: pcmk_cpg_membership: Member[0.0] cib.3232261517 
Nov 13 13:44:20 vm2 cib[31459]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:20 vm2 cib[31459]:     info: pcmk_cpg_membership: Member[0.1] cib.3232261518 
Nov 13 13:44:20 vm2 cib[31465]:     info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: 978cb58a57d1ff0f3e53e793331143d7)
Nov 13 13:44:20 vm2 cib[31465]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.TidZSC (digest: /var/lib/pacemaker/cib/cib.g2LB8U)
Nov 13 13:44:21 vm2 cib[31459]:     info: pcmk_cpg_membership: Joined[1.0] cib.3232261519 
Nov 13 13:44:21 vm2 cib[31459]:     info: pcmk_cpg_membership: Member[1.0] cib.3232261517 
Nov 13 13:44:21 vm2 cib[31459]:     info: pcmk_cpg_membership: Member[1.1] cib.3232261518 
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_get_peer: Created entry ac162f3c-00db-47b5-97aa-07743db1cf87/0xbcc370 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm2 cib[31459]:     info: pcmk_cpg_membership: Member[1.2] cib.3232261519 
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_client_new: Connecting 0xbcc3e0 for uid=496 gid=492 pid=31464 id=75bfca14-389a-44dd-baa6-e2249c558c46
Nov 13 13:44:21 vm2 crmd[31464]:     info: do_cib_control: CIB connection established
Nov 13 13:44:21 vm2 crmd[31464]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
Nov 13 13:44:21 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.0.0)
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Created entry fa843376-669b-440e-b3be-9e1b531811cf/0x22a4c50 for node (null)/3232261518 (1 total)
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261518 has uuid 3232261518
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[3232261518] - corosync-cpg is now online
Nov 13 13:44:21 vm2 crmd[31464]:     info: init_cs_connection_once: Connection to 'corosync': established
Nov 13 13:44:21 vm2 crmd[31464]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:21 vm2 crmd[31464]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261518 is now known as vm2
Nov 13 13:44:21 vm2 crmd[31464]:     info: peer_update_callback: vm2 is now (null)
Nov 13 13:44:21 vm2 crmd[31464]:   notice: cluster_connect_quorum: Quorum acquired
Nov 13 13:44:21 vm2 crmd[31464]:     info: do_ha_control: Connected to the cluster
Nov 13 13:44:21 vm2 crmd[31464]:     info: lrmd_ipc_connect: Connecting to lrmd
Nov 13 13:44:21 vm2 cib[31459]:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/3, version=0.0.0)
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_client_new: Connecting 0xa1abe0 for uid=496 gid=492 pid=31462 id=d0c3e73b-a852-467e-b374-c879c3a17325
Nov 13 13:44:21 vm2 lrmd[31461]:     info: crm_client_new: Connecting 0x9a1c10 for uid=496 gid=492 pid=31464 id=a8bb49c2-413e-47b0-9c1f-7223841526a5
Nov 13 13:44:21 vm2 crmd[31464]:     info: do_lrm_control: LRM connection established
Nov 13 13:44:21 vm2 crmd[31464]:     info: do_started: Delaying start, no membership data (0000000000100000)
Nov 13 13:44:21 vm2 crmd[31464]:     info: pcmk_quorum_notification: Membership 12: quorum retained (3)
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Created entry 985ec8e9-63b0-489b-a16f-50c5e51157f8/0x23e9520 for node (null)/3232261517 (2 total)
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:21 vm2 crmd[31464]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261517
Nov 13 13:44:21 vm2 cib[31459]:     info: crm_client_new: Connecting 0xc4d380 for uid=0 gid=0 pid=31460 id=33d0d3e6-9fa9-402f-8c59-704260f2b8eb
Nov 13 13:44:21 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/4, version=0.0.0)
Nov 13 13:44:21 vm2 attrd[31462]:     info: attrd_cib_connect: Connected to the CIB after 2 attempts
Nov 13 13:44:21 vm2 attrd[31462]:     info: main: CIB connection active
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Joined[0.0] attrd.3232261518 
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_get_peer: Created entry a03a649d-0e44-4422-9c29-a7150e3a3abd/0x876f00 for node (null)/3232261517 (2 total)
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Member[0.0] attrd.3232261517 
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:21 vm2 attrd[31462]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261517] - state is now member (was (null))
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Member[0.1] attrd.3232261518 
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Joined[1.0] attrd.3232261519 
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Member[1.0] attrd.3232261517 
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Member[1.1] attrd.3232261518 
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_get_peer: Created entry 24830da6-ef8e-47a4-ba48-af3015946da6/0x876f70 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm2 attrd[31462]:     info: pcmk_cpg_membership: Member[1.2] attrd.3232261519 
Nov 13 13:44:21 vm2 attrd[31462]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:21 vm2 attrd[31462]:   notice: crm_update_peer_state: attrd_peer_change_cb: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:21 vm2 stonith-ng[31460]:   notice: setup_cib: Watching for stonith topology changes
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: qb_ipcs_us_publish: server name: stonith-ng
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: main: Starting stonith-ng mainloop
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Joined[0.0] stonith-ng.3232261518 
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: crm_get_peer: Created entry 2fc2a3ca-4e97-4f23-9299-b0de2ef1d581/0xfed790 for node (null)/3232261517 (2 total)
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261517 has uuid 3232261517
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Member[0.0] stonith-ng.3232261517 
Nov 13 13:44:21 vm2 stonith-ng[31460]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:21 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.0.0)
Nov 13 13:44:21 vm2 crmd[31464]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261517
Nov 13 13:44:21 vm2 crmd[31464]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261517] - state is now member (was (null))
Nov 13 13:44:21 vm2 crmd[31464]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node vm2[3232261518] - state is now member (was (null))
Nov 13 13:44:21 vm2 crmd[31464]:     info: peer_update_callback: vm2 is now member (was (null))
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Created entry 356c3337-e8fa-4139-8e23-ed1f23d8a87e/0x23ebdb0 for node (null)/3232261519 (3 total)
Nov 13 13:44:21 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:21 vm2 crmd[31464]:     info: pcmk_quorum_notification: Obtaining name for new node 3232261519
Nov 13 13:44:22 vm2 stonith-ng[31460]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:22 vm2 stonith-ng[31460]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Member[0.1] stonith-ng.3232261518 
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: init_cib_cache_cb: Updating device list from the cib: init
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: unpack_nodes: Creating a fake local node
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Joined[1.0] stonith-ng.3232261519 
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Member[1.0] stonith-ng.3232261517 
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Member[1.1] stonith-ng.3232261518 
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: crm_get_peer: Created entry d625e56e-67d3-4188-b48e-b500c95d4cbb/0xff2770 for node (null)/3232261519 (3 total)
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261519 has uuid 3232261519
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: pcmk_cpg_membership: Member[1.2] stonith-ng.3232261519 
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:22 vm2 crmd[31464]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261519
Nov 13 13:44:22 vm2 crmd[31464]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[3232261519] - state is now member (was (null))
Nov 13 13:44:22 vm2 crmd[31464]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:22 vm2 crmd[31464]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:22 vm2 crmd[31464]:     info: do_started: Delaying start, Config not read (0000000000000040)
Nov 13 13:44:22 vm2 crmd[31464]:     info: qb_ipcs_us_publish: server name: crmd
Nov 13 13:44:22 vm2 crmd[31464]:   notice: do_started: The local CRM is operational
Nov 13 13:44:22 vm2 crmd[31464]:     info: do_log: FSA: Input I_PENDING from do_started() received in state S_STARTING
Nov 13 13:44:22 vm2 crmd[31464]:   notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Nov 13 13:44:22 vm2 cib[31459]:     info: cib_process_request: Completed cib_slave operation for section 'all': OK (rc=0, origin=local/crmd/5, version=0.0.0)
Nov 13 13:44:22 vm2 stonith-ng[31460]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Joined[0.0] crmd.3232261518 
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Member[0.0] crmd.3232261517 
Nov 13 13:44:23 vm2 crmd[31464]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261517] - corosync-cpg is now online
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Member[0.1] crmd.3232261518 
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Joined[1.0] crmd.3232261519 
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Member[1.0] crmd.3232261517 
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Member[1.1] crmd.3232261518 
Nov 13 13:44:23 vm2 crmd[31464]:     info: pcmk_cpg_membership: Member[1.2] crmd.3232261519 
Nov 13 13:44:23 vm2 crmd[31464]:     info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3232261519] - corosync-cpg is now online
Nov 13 13:44:23 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:23 vm2 crmd[31464]:     info: peer_update_callback: vm3 is now member
Nov 13 13:44:23 vm2 crmd[31464]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:23 vm2 crmd[31464]:     info: peer_update_callback: vm1 is now member
Nov 13 13:44:24 vm2 stonith-ng[31460]:     info: crm_client_new: Connecting 0xff4900 for uid=496 gid=492 pid=31464 id=d20e141c-69f8-4300-ba16-268f3c5cae0c
Nov 13 13:44:24 vm2 stonith-ng[31460]:     info: stonith_command: Processed register from crmd.31464: OK (0)
Nov 13 13:44:24 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify from crmd.31464: OK (0)
Nov 13 13:44:24 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify from crmd.31464: OK (0)
Nov 13 13:44:42 vm2 crmd[31464]:     info: election_count_vote: Election 1 (owner: 3232261517) lost: vote from vm1 (Uptime)
Nov 13 13:44:42 vm2 crmd[31464]:     info: do_log: FSA: Input I_PENDING from do_election_count_vote() received in state S_PENDING
Nov 13 13:44:43 vm2 cib[31459]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:43 vm2 cib[31459]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:43 vm2 cib[31459]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section cib: OK (rc=0, origin=vm1/crmd/7, version=0.0.1)
Nov 13 13:44:43 vm2 crmd[31464]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:44:43 vm2 crmd[31464]:     info: election_count_vote: Election 2 (owner: 3232261517) lost: vote from vm1 (Uptime)
Nov 13 13:44:43 vm2 crmd[31464]:     info: update_dc: Unset DC. Was vm1
Nov 13 13:44:43 vm2 crmd[31464]:     info: do_log: FSA: Input I_PENDING from do_election_count_vote() received in state S_PENDING
Nov 13 13:44:43 vm2 crmd[31464]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section crm_config: OK (rc=0, origin=vm1/crmd/9, version=0.1.1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/6, version=0.1.1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/7, version=0.1.1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/8, version=0.1.1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section crm_config: OK (rc=0, origin=vm1/crmd/11, version=0.2.1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/9, version=0.2.1)
Nov 13 13:44:43 vm2 cib[31473]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_replace: Digest matched on replace from vm1: 360fde11e7cf93696f974eea17cffd9b
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_replace: Replaced 0.2.1 with 0.2.1 from vm1
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=vm1/crmd/17, version=0.2.1)
Nov 13 13:44:43 vm2 crmd[31464]:     info: erase_status_tag: Deleting xpath: //node_state[@uname='vm2']/transient_attributes
Nov 13 13:44:43 vm2 crmd[31464]:     info: update_attrd_helper: Connecting to attrd... 5 retries remaining
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section nodes: OK (rc=0, origin=vm1/crmd/18, version=0.3.1)
Nov 13 13:44:43 vm2 attrd[31462]:     info: crm_client_new: Connecting 0x874000 for uid=496 gid=492 pid=31464 id=f389c2a1-b596-496a-9b25-1c3be7932149
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section nodes: OK (rc=0, origin=vm1/crmd/19, version=0.4.1)
Nov 13 13:44:43 vm2 cib[31473]:     info: write_cib_contents: Wrote version 0.2.0 of the CIB to disk (digest: 7c397f6c57041145e23f3494e809aec1)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='vm2']/transient_attributes to master (origin=local/crmd/10)
Nov 13 13:44:43 vm2 cib[31473]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.4OH9Nt (digest: /var/lib/pacemaker/cib/cib.PwoEkK)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section nodes: OK (rc=0, origin=vm1/crmd/20, version=0.5.1)
Nov 13 13:44:43 vm2 crmd[31464]:     info: do_log: FSA: Input I_NOT_DC from do_cl_join_finalize_respond() received in state S_PENDING
Nov 13 13:44:43 vm2 crmd[31464]:   notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_client_message: Starting an election to determine the writer
Nov 13 13:44:43 vm2 cib[31459]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:43 vm2 cib[31474]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/23, version=0.5.2)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: corosync_node_name: Unable to get node name for nodeid 3232261518
Nov 13 13:44:43 vm2 attrd[31462]:   notice: get_node_name: Defaulting to uname -n for the local corosync node name
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_client_message: Broadcasting terminate[vm2] = (null)
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_client_message: Broadcasting shutdown[vm2] = (null)
Nov 13 13:44:43 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261517 is now known as vm1
Nov 13 13:44:43 vm2 attrd[31462]:     info: election_count_vote: Election 1 (owner: 3232261517) pass: vote from vm1 (Uptime)
Nov 13 13:44:43 vm2 attrd[31462]:     info: crm_get_peer: Node 3232261519 is now known as vm3
Nov 13 13:44:43 vm2 attrd[31462]:     info: election_count_vote: Election 1 (owner: 3232261519) pass: vote from vm3 (Uptime)
Nov 13 13:44:43 vm2 attrd[31462]:     info: election_count_vote: Election 2 (owner: 3232261517) pass: vote from vm1 (Uptime)
Nov 13 13:44:43 vm2 cib[31474]:     info: write_cib_contents: Wrote version 0.5.0 of the CIB to disk (digest: 630d79f602055b52fd2ea79fdbd1baf8)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/25, version=0.5.3)
Nov 13 13:44:43 vm2 cib[31474]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.29KSZA (digest: /var/lib/pacemaker/cib/cib.pyh1KR)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/27, version=0.5.4)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section cib: OK (rc=0, origin=vm1/crmd/30, version=0.5.5)
Nov 13 13:44:43 vm2 attrd[31462]:     info: election_timer_cb: Election election-attrd complete
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/2)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: write_attribute: Sent update 2 with 3 changes for shutdown, id=<n/a>, set=(null)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/3)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: write_attribute: Sent update 3 with 3 changes for terminate, id=<n/a>, set=(null)
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_client_message: Broadcasting probe_complete[vm2] = true (writer)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: write_attribute: Sent update 4 with 1 changes for probe_complete, id=<n/a>, set=(null)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/4)
Nov 13 13:44:43 vm2 attrd[31462]:     info: write_attribute: Write out of probe_complete delayed: update 4 in progress
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/attrd/2, version=0.5.6)
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_cib_callback: Update 2 for shutdown: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 2 for shutdown[vm1]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 2 for shutdown[vm2]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 2 for shutdown[vm3]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:     info: write_attribute: Write out of probe_complete delayed: update 4 in progress
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_cib_callback: Update 3 for terminate: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 3 for terminate[vm1]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 3 for terminate[vm2]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 3 for terminate[vm3]=(null): OK (0)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/attrd/4, version=0.5.7)
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_cib_callback: Update 4 for probe_complete: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 4 for probe_complete[vm1]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 4 for probe_complete[vm2]=(null): OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 4 for probe_complete[vm3]=true: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: write_attribute: Sent update 5 with 3 changes for probe_complete, id=<n/a>, set=(null)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/5)
Nov 13 13:44:43 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/attrd/5, version=0.5.8)
Nov 13 13:44:43 vm2 attrd[31462]:     info: attrd_cib_callback: Update 5 for probe_complete: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 5 for probe_complete[vm1]=true: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 5 for probe_complete[vm2]=true: OK (0)
Nov 13 13:44:43 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 5 for probe_complete[vm3]=true: OK (0)
Nov 13 13:44:51 vm2 crmd[31464]:     info: throttle_send_command: Updated throttle state to 0000
Nov 13 13:45:33 vm2 crmd[31464]:     info: election_count_vote: Election 3 (owner: 3232261517) lost: vote from vm1 (Uptime)
Nov 13 13:45:33 vm2 crmd[31464]:     info: update_dc: Unset DC. Was vm1
Nov 13 13:45:33 vm2 crmd[31464]:     info: do_log: FSA: Input I_PENDING from do_election_count_vote() received in state S_NOT_DC
Nov 13 13:45:33 vm2 crmd[31464]:   notice: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_slave operation for section 'all': OK (rc=0, origin=local/crmd/11, version=0.5.8)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=vm1/cibadmin/2, version=0.6.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/12, version=0.6.1)
Nov 13 13:45:33 vm2 stonith-ng[31460]:     info: stonith_level_remove: Node vm3 not found (0 active entries)
Nov 13 13:45:33 vm2 stonith-ng[31460]:     info: stonith_level_register: Node vm3 has 1 active fencing levels
Nov 13 13:45:33 vm2 stonith-ng[31460]:     info: update_cib_stonith_devices: Updating device list from the cib: new resource
Nov 13 13:45:33 vm2 stonith-ng[31460]:  warning: handle_startup_fencing: Blind faith: not fencing unseen nodes
Nov 13 13:45:33 vm2 stonith-ng[31460]:     info: cib_device_update: Device F1 is allowed on vm2: score=100
Nov 13 13:45:33 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action metadata for agent fence_legacy (target=(null))
Nov 13 13:45:33 vm2 crmd[31464]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:45:33 vm2 crmd[31464]:     info: election_count_vote: Election 4 (owner: 3232261517) lost: vote from vm1 (Uptime)
Nov 13 13:45:33 vm2 crmd[31464]:     info: update_dc: Unset DC. Was vm1
Nov 13 13:45:33 vm2 crmd[31464]:     info: do_log: FSA: Input I_PENDING from do_election_count_vote() received in state S_PENDING
Nov 13 13:45:33 vm2 crmd[31464]:     info: update_dc: Set DC to vm1 (3.0.8)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section crm_config: OK (rc=0, origin=vm1/crmd/39, version=0.7.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section crm_config: OK (rc=0, origin=vm1/crmd/41, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/13, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/14, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/15, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/16, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_replace: Digest matched on replace from vm1: b65668c649a0f8a465a42db6c017bc19
Nov 13 13:45:33 vm2 crmd[31464]:     info: do_log: FSA: Input I_NOT_DC from do_cl_join_finalize_respond() received in state S_PENDING
Nov 13 13:45:33 vm2 crmd[31464]:   notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_replace: Replaced 0.8.1 with 0.8.1 from vm1
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=vm1/crmd/47, version=0.8.1)
Nov 13 13:45:33 vm2 cib[31482]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw
Nov 13 13:45:33 vm2 cib[31482]:     info: write_cib_contents: Wrote version 0.6.0 of the CIB to disk (digest: 2db643db6cb3c3f1825600265949deb4)
Nov 13 13:45:33 vm2 cib[31482]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.3jZz9s (digest: /var/lib/pacemaker/cib/cib.c9VpRX)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section //node_state[@uname='vm3']/lrm: OK (rc=0, origin=vm1/crmd/51, version=0.8.2)
Nov 13 13:45:33 vm2 cib[31484]:     info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/52, version=0.8.3)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section //node_state[@uname='vm1']/lrm: OK (rc=0, origin=vm1/crmd/53, version=0.8.4)
Nov 13 13:45:33 vm2 cib[31484]:     info: write_cib_contents: Wrote version 0.8.0 of the CIB to disk (digest: 9db35554f5ac4e48336f1bae33d89abc)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/54, version=0.8.5)
Nov 13 13:45:33 vm2 cib[31484]:     info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.GKA9cx (digest: /var/lib/pacemaker/cib/cib.XzSBg2)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section //node_state[@uname='vm2']/lrm: OK (rc=0, origin=vm1/crmd/55, version=0.8.6)
Nov 13 13:45:33 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/56, version=0.8.7)
Nov 13 13:45:34 vm2 stonith-ng[31460]:   notice: stonith_device_register: Added 'F1' to the device list (1 active devices)
Nov 13 13:45:35 vm2 lrmd[31461]:     info: process_lrmd_get_rsc_info: Resource 'F1' not found (0 active resources)
Nov 13 13:45:35 vm2 lrmd[31461]:     info: process_lrmd_rsc_register: Added 'F1' to the rsc list (1 active resources)
Nov 13 13:45:35 vm2 crmd[31464]:     info: do_lrm_rsc_op: Performing key=7:1:7:154fb289-24e8-407e-9a03-69a510480b60 op=F1_monitor_0
Nov 13 13:45:35 vm2 stonith-ng[31460]:     info: crm_client_new: Connecting 0x103cfc0 for uid=0 gid=0 pid=31461 id=5ac4cd86-62c4-4dd0-83b0-0320ceb1dac6
Nov 13 13:45:35 vm2 stonith-ng[31460]:     info: stonith_command: Processed register from lrmd.31461: OK (0)
Nov 13 13:45:35 vm2 lrmd[31461]:     info: process_lrmd_get_rsc_info: Resource 'pDummy' not found (1 active resources)
Nov 13 13:45:35 vm2 lrmd[31461]:     info: process_lrmd_rsc_register: Added 'pDummy' to the rsc list (2 active resources)
Nov 13 13:45:35 vm2 crmd[31464]:     info: do_lrm_rsc_op: Performing key=8:1:7:154fb289-24e8-407e-9a03-69a510480b60 op=pDummy_monitor_0
Nov 13 13:45:35 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify from lrmd.31461: OK (0)
Nov 13 13:45:35 vm2 Dummy(pDummy)[31485]: DEBUG: pDummy monitor : 7
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/17)
Nov 13 13:45:36 vm2 crmd[31464]:     info: process_lrm_event: LRM operation F1_monitor_0 (call=5, rc=7, cib-update=17, confirmed=true) not running
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/17, version=0.8.8)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/61, version=0.8.9)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/17, version=0.8.10)
Nov 13 13:45:36 vm2 crmd[31464]:     info: services_os_action_execute: Managed Dummy_meta-data_0 process 31506 exited with rc=0
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/18)
Nov 13 13:45:36 vm2 crmd[31464]:   notice: process_lrm_event: LRM operation pDummy_monitor_0 (call=9, rc=7, cib-update=18, confirmed=true) not running
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/18, version=0.8.11)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/62, version=0.8.12)
Nov 13 13:45:36 vm2 attrd[31462]:     info: attrd_client_message: Broadcasting probe_complete[vm2] = true (writer)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/18, version=0.8.13)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/19, version=0.8.14)
Nov 13 13:45:36 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/20, version=0.8.15)
Nov 13 13:45:38 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/63, version=0.8.16)
Nov 13 13:45:39 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/64, version=0.8.17)
Nov 13 13:47:06 vm2 attrd[31462]:   notice: write_attribute: Sent update 6 with 1 changes for fail-count-pDummy, id=<n/a>, set=(null)
Nov 13 13:47:06 vm2 attrd[31462]:   notice: write_attribute: Sent update 7 with 1 changes for last-failure-pDummy, id=<n/a>, set=(null)
Nov 13 13:47:06 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/crmd/21, version=0.8.18)
Nov 13 13:47:06 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/6)
Nov 13 13:47:06 vm2 cib[31459]:     info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/7)
Nov 13 13:47:06 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/attrd/6, version=0.8.19)
Nov 13 13:47:06 vm2 attrd[31462]:     info: attrd_cib_callback: Update 6 for fail-count-pDummy: OK (0)
Nov 13 13:47:06 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 6 for fail-count-pDummy[vm3]=1: OK (0)
Nov 13 13:47:06 vm2 cib[31459]:     info: cib_process_request: Completed cib_apply_diff operation for section status: OK (rc=0, origin=vm1/attrd/7, version=0.8.20)
Nov 13 13:47:06 vm2 attrd[31462]:     info: attrd_cib_callback: Update 7 for last-failure-pDummy: OK (0)
Nov 13 13:47:06 vm2 attrd[31462]:   notice: attrd_cib_callback: Update 7 for last-failure-pDummy[vm3]=1384318026: OK (0)
Nov 13 13:47:08 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:08 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action list for agent fence_legacy (target=(null))
Nov 13 13:47:08 vm2 stonith-ng[31460]:     info: dynamic_list_search_cb: Refreshing port list for F1
Nov 13 13:47:08 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:08 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:09 vm2 stonith: [31571]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:09 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:12 vm2 stonith: [31583]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:12 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:12 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31582] (call 2 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:12 vm2 stonith-ng[31460]:  warning: log_operation: F1:31582 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:12 vm2 stonith-ng[31460]:  warning: log_operation: F1:31582 [ failed: vm3 5 ]
Nov 13 13:47:12 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.696fb2c3: Generic Pacemaker error
Nov 13 13:47:12 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=696fb2c3-e11a-4124-ba9b-bafc9ab28426) by client crmd.15883
Nov 13 13:47:12 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:14 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:14 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:14 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:14 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:15 vm2 stonith: [31595]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:15 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:17 vm2 stonith: [31607]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:17 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:17 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31606] (call 3 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:17 vm2 stonith-ng[31460]:  warning: log_operation: F1:31606 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:17 vm2 stonith-ng[31460]:  warning: log_operation: F1:31606 [ failed: vm3 5 ]
Nov 13 13:47:17 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.431c7488: Generic Pacemaker error
Nov 13 13:47:17 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=431c7488-013e-4900-bde7-a3ce154b35a3) by client crmd.15883
Nov 13 13:47:17 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:19 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:19 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:19 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:19 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:20 vm2 stonith: [31619]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:20 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:22 vm2 stonith: [31631]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:22 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:22 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31630] (call 4 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:22 vm2 stonith-ng[31460]:  warning: log_operation: F1:31630 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:22 vm2 stonith-ng[31460]:  warning: log_operation: F1:31630 [ failed: vm3 5 ]
Nov 13 13:47:22 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.682bdc12: Generic Pacemaker error
Nov 13 13:47:22 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=682bdc12-35a4-431a-8773-4862cc8c39ef) by client crmd.15883
Nov 13 13:47:22 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:24 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:24 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:24 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:24 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:25 vm2 stonith: [31643]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:25 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:27 vm2 stonith: [31655]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:27 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:27 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31654] (call 5 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:27 vm2 stonith-ng[31460]:  warning: log_operation: F1:31654 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:27 vm2 stonith-ng[31460]:  warning: log_operation: F1:31654 [ failed: vm3 5 ]
Nov 13 13:47:27 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.d761e73f: Generic Pacemaker error
Nov 13 13:47:27 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=d761e73f-f337-48cc-b2a1-5b2d722d2738) by client crmd.15883
Nov 13 13:47:27 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:29 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:29 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:33 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.11df91ab: Generic Pacemaker error
Nov 13 13:47:33 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:33 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=11df91ab-fc81-43aa-941d-ffa1204df1c9) by client crmd.15883
Nov 13 13:47:35 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:35 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:35 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:35 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:36 vm2 stonith: [31673]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:36 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:38 vm2 stonith: [31685]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:38 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:38 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31684] (call 7 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:38 vm2 stonith-ng[31460]:  warning: log_operation: F1:31684 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:38 vm2 stonith-ng[31460]:  warning: log_operation: F1:31684 [ failed: vm3 5 ]
Nov 13 13:47:38 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.84777767: Generic Pacemaker error
Nov 13 13:47:38 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=84777767-aa8b-4e04-8dec-b26dae36aaff) by client crmd.15883
Nov 13 13:47:38 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:40 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:40 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:40 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:40 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:41 vm2 stonith: [31697]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:41 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:43 vm2 stonith: [31709]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:43 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:43 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31708] (call 8 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:43 vm2 stonith-ng[31460]:  warning: log_operation: F1:31708 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:43 vm2 stonith-ng[31460]:  warning: log_operation: F1:31708 [ failed: vm3 5 ]
Nov 13 13:47:43 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.588ca7d3: Generic Pacemaker error
Nov 13 13:47:43 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=588ca7d3-cb01-4e1b-9c7d-5fcdd5b66a27) by client crmd.15883
Nov 13 13:47:43 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:45 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:45 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:45 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:45 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:46 vm2 stonith: [31721]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:46 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:48 vm2 stonith: [31733]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:48 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:48 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31732] (call 9 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:48 vm2 stonith-ng[31460]:  warning: log_operation: F1:31732 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:48 vm2 stonith-ng[31460]:  warning: log_operation: F1:31732 [ failed: vm3 5 ]
Nov 13 13:47:48 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.a3379e0c: Generic Pacemaker error
Nov 13 13:47:48 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=a3379e0c-d206-4ced-9e7e-1c915f08a0ae) by client crmd.15883
Nov 13 13:47:48 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:50 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:50 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:50 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:50 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:51 vm2 stonith: [31745]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:51 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:54 vm2 stonith: [31757]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:54 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:54 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31756] (call 10 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:54 vm2 stonith-ng[31460]:  warning: log_operation: F1:31756 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:54 vm2 stonith-ng[31460]:  warning: log_operation: F1:31756 [ failed: vm3 5 ]
Nov 13 13:47:54 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.9ab4c26b: Generic Pacemaker error
Nov 13 13:47:54 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=9ab4c26b-da3e-40cd-ba98-c89017db4953) by client crmd.15883
Nov 13 13:47:54 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:47:56 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:47:56 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:47:56 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:47:56 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:47:57 vm2 stonith: [31769]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:57 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:47:59 vm2 stonith: [31781]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:47:59 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:47:59 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31780] (call 11 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:47:59 vm2 stonith-ng[31460]:  warning: log_operation: F1:31780 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:47:59 vm2 stonith-ng[31460]:  warning: log_operation: F1:31780 [ failed: vm3 5 ]
Nov 13 13:47:59 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.1ba836f2: Generic Pacemaker error
Nov 13 13:47:59 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=1ba836f2-328d-45c7-adbb-1db9b0a1ca4c) by client crmd.15883
Nov 13 13:47:59 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 13:48:01 vm2 stonith-ng[31460]:   notice: can_fence_host_with_device: F1 can fence vm3: dynamic-list
Nov 13 13:48:01 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 13:48:01 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_fence from vm1: Operation now in progress (-115)
Nov 13 13:48:01 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action reboot for agent fence_legacy (target=vm3)
Nov 13 13:48:02 vm2 stonith: [31793]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:48:02 vm2 stonith-ng[31460]:     info: internal_stonith_action_execute: Attempt 2 to execute fence_legacy (reboot). remaining timeout is 59
Nov 13 13:48:04 vm2 stonith: [31825]: CRIT: external_reset_req: 'libvirt reset' for host vm3 failed with rc 1
Nov 13 13:48:04 vm2 stonith-ng[31460]:     info: update_remaining_timeout: Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
Nov 13 13:48:04 vm2 stonith-ng[31460]:    error: log_operation: Operation 'reboot' [31823] (call 12 from crmd.15883) for host 'vm3' with device 'F1' returned: -201 (Generic Pacemaker error)
Nov 13 13:48:04 vm2 stonith-ng[31460]:  warning: log_operation: F1:31823 [ Performing: stonith -t external/libvirt -T reset vm3 ]
Nov 13 13:48:04 vm2 stonith-ng[31460]:  warning: log_operation: F1:31823 [ failed: vm3 5 ]
Nov 13 13:48:04 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.00825b71: Generic Pacemaker error
Nov 13 13:48:04 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=00825b71-24e3-4f14-a0b8-6945f050dfd1) by client crmd.15883
Nov 13 13:48:04 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
Nov 13 14:03:04 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_query from vm1: OK (0)
Nov 13 14:03:04 vm2 stonith-ng[31460]:     info: stonith_action_create: Initiating action list for agent fence_legacy (target=(null))
Nov 13 14:03:04 vm2 stonith-ng[31460]:     info: dynamic_list_search_cb: Refreshing port list for F1
Nov 13 14:03:07 vm2 stonith-ng[31460]:   notice: remote_op_done: Operation reboot of vm3 by vm1 for crmd.15883@vm1.893bcd8c: Generic Pacemaker error
Nov 13 14:03:07 vm2 crmd[31464]:   notice: tengine_stonith_notify: Peer vm3 was not terminated (reboot) by vm1 for vm1: Generic Pacemaker error (ref=893bcd8c-11ea-4f2c-b5d5-e2c9d3883c1b) by client crmd.15883
Nov 13 14:03:07 vm2 stonith-ng[31460]:     info: stonith_command: Processed st_notify reply from vm1: OK (0)
