[Pacemaker] Problem on Pacemaker when a new node is joining the cluster

Paul Sun paul.sun at syniverse.com
Tue Mar 12 03:41:10 UTC 2013 

Hi

I have 3 nodes setup, and I have configured IP and LSB service. The LSB service is relied on IP service. Given that I have startup node1 and node2, and then node3 is started to join the cluster, it is found that the LSB service is restarted on node2, is there any reason? I observed "Sending flush op to all hosts for: probe_complete (true)" is being initiated, why?

Below is my configuration.

crm(live)configure# show
node HA_NODE1
node HA_NODE2
node HA_NODE3
primitive ClusterIP ocf:heartbeat:IPaddr2 \
      params ip="192.168.206.161" cidr_netmask="32" \
      op monitor interval="30s" \
      meta resource-stickiness="100"
primitive PDNS_D lsb:pdnsd \
      op monitor interval="30s" \
      meta resource-stickiness="1000"
primitive PDNS_IP ocf:heartbeat:IPaddr2 \
      params ip="192.168.206.160" cidr_netmask="32" nic="eth2" \
      op monitor interval="30s" \
      meta resource-stickiness="1000"
location ClusterIP-prefer1 ClusterIP 500: HA_NODE1
location ClusterIP-prefer2 ClusterIP 300: HA_NODE2
location ClusterIP-prefer3 ClusterIP -inf: HA_NODE3
location PDNS_D-prefer1 PDNS_D -inf: HA_NODE1
location PDNS_IP-prefer1 PDNS_IP -inf: HA_NODE1
location PDNS_IP-prefer2 PDNS_IP 500: HA_NODE2
location PDNS_IP-prefer3 PDNS_IP 300: HA_NODE3
colocation PDNS_Services inf: PDNS_D PDNS_IP
order PDNS_Order inf: PDNS_IP PDNS_D
property $id="cib-bootstrap-options" \
      dc-version="1.1.8-7.el6-394e906" \
      cluster-infrastructure="classic openais (with plugin)" \
      expected-quorum-votes="3" \
      stonith-enabled="false" \
      no-quorum-policy="ignore" \
      last-lrm-refresh="1362737116"


Log from node2.
===============
Mar 12 11:39:11 HA_NODE2 crmd[31080]:   notice: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_JOIN_OFFER cause=C_HA_MESSAGE origin=route_message ]
Mar 12 11:39:12 HA_NODE2 cib[31075]:  warning: cib_process_diff: Diff 0.76.0 -> 0.76.1 from HA_NODE3 not applied to 0.76.78: current "num_updates" is greater than required
Mar 12 11:39:15 HA_NODE2 attrd[31078]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 12 11:39:15 HA_NODE2 attrd[31078]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 12 11:39:15 HA_NODE2 crmd[31080]:   notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 12 11:39:17 HA_NODE2 crmd[31080]:   notice: process_lrm_event: LRM operation PDNS_D_stop_0 (call=45, rc=0, cib-update=27, confirmed=true) ok
Mar 12 11:39:27 HA_NODE2 pdns[32333]: PowerDNS recursor 3.3 (C) 2001-2010 PowerDNS.COM BV (Jan 31 2013, 18:07:25, gcc 4.1.2 20080704 (Red Hat 4.1.2-51)) starting up
Mar 12 11:39:27 HA_NODE2 pdns[32333]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Operating in 64 bits mode
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Reading random entropy from '/dev/urandom'
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Only allowing queries from: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10, 0.0.0.0, ::
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Redirecting queries for zone 'e164.arpa.' to: 192.168.209.242:53
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Inserting rfc 1918 private space zones
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Not decreasing socket buffer size from 229376 to 200000
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Listening for UDP queries on 192.168.206.160:5300
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Enabled TCP data-ready filter for (slight) DoS protection
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Listening for TCP queries on 192.168.206.160:5300
Mar 12 11:39:27 HA_NODE2 pdns[32333]: Calling daemonize, going to background
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Unable to parse configuration file '/etc/powerdns/recursor.conf' ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 PowerDNS recursor 3.3 (C) 2001-2010 PowerDNS.COM BV (Jan 31 2013, 18:07:25, gcc 4.1.2 20080704 (Red Hat 4.1.2-51)) starting up ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2. ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Operating in 64 bits mode ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Reading random entropy from '/dev/urandom' ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Only allowing queries from: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10 ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10, 0.0.0.0, :: ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Redirecting queries for zone 'e164.arpa.' to: 192.168.209.242:53 ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Inserting rfc 1918 private space zones ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Not decreasing socket buffer size from 229376 to 200000 ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Listening for UDP queries on 192.168.206.160:5300 ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Enabled TCP data-ready filter for (slight) DoS protection ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Listening for TCP queries on 192.168.206.160:5300 ]
Mar 12 11:39:27 HA_NODE2 lrmd[31077]:   notice: operation_finished: PDNS_D_start_0:32323 [ Mar 12 11:39:27 Calling daemonize, going to background ]
Mar 12 11:39:27 HA_NODE2 crmd[31080]:   notice: process_lrm_event: LRM operation PDNS_D_start_0 (call=49, rc=0, cib-update=28, confirmed=true) ok
Mar 12 11:39:27 HA_NODE2 pdns[32334]: Launching 2 threads
Mar 12 11:39:27 HA_NODE2 pdns[32334]: Done priming cache with root hints
Mar 12 11:39:27 HA_NODE2 pdns[32334]: Done priming cache with root hints
Mar 12 11:39:27 HA_NODE2 pdns[32334]: Enabled 'epoll' multiplexer
Mar 12 11:39:27 HA_NODE2 crmd[31080]:   notice: process_lrm_event: LRM operation PDNS_D_monitor_30000 (call=52, rc=0, cib-update=29, confirmed=false) ok

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130312/8c1fc74b/attachment-0003.html>

More information about the Pacemaker mailing list