[Pacemaker] Need help with resolving very long election cycle

Thu Feb 2 22:31:52 UTC 2012

On Thu, Feb 2, 2012 at 9:55 PM, Shyam <shyam.kaushik at gmail.com> wrote:
> Hi Andrew,
>
> Here is more logs covering a larger period that shows multiple of this
> election cycle. Please note that in the below case I had set dc-deadtime to
> 5secs & the I_DC_TIMEOUT pops up every 5 secs. I turned this dc-deadtime to
> 10secs & the long election cycle problem disappeared. It no longer happens.
> I suspect that before a single election cycle completes, the next
> I_DC_TIMEOUT kicks-in. Could this be the reason?

Yes.  The question is why the cycle is taking so long :-/

>
> On node 1:
> Jan 17 12:00:02 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 3 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:02 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:02 vsa-0000003ca-vc-0 crmd: [1120]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:02 vsa-0000003ca-vc-0 crmd: [1120]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:03 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 4 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:03 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 5 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:04 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Jan 17 12:00:04 vsa-0000003ca-vc-0 crmd: [1120]: info: start_subsystem:
> Starting sub-system "pengine"
> Jan 17 12:00:04 vsa-0000003ca-vc-0 crmd: [1120]: WARN: start_subsystem:
> Client pengine already running as pid 4243
> Jan 17 12:00:08 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_takeover:
> Taking over DC status for this partition
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/O mode
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/108, version=1.8.26): ok (rc=0)
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/W mode
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/109,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/110,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/112, version=1.8.26): ok (rc=0)
> Jan 17 12:00:08 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_join_offer_all:
> join-5: Waiting on 2 outstanding join acks
> Jan 17 12:00:08 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/114, version=1.8.26): ok (rc=0)
> Jan 17 12:00:08 vsa-0000003ca-vc-0 crmd: [1120]: info:
> config_query_callback: Checking for expired actions every 900000ms
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 6 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: info: update_dc: Set DC to
> vsa-0000003ca-vc-0 (3.0.1)
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: info: update_dc: Unset DC
> vsa-0000003ca-vc-0
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 7 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:09 vsa-0000003ca-vc-0 crmd: [1120]: WARN: do_log: FSA: Input
> I_JOIN_REQUEST from route_message() received in state S_ELECTION
> Jan 17 12:00:10 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Jan 17 12:00:10 vsa-0000003ca-vc-0 crmd: [1120]: info: start_subsystem:
> Starting sub-system "pengine"
> Jan 17 12:00:10 vsa-0000003ca-vc-0 crmd: [1120]: WARN: start_subsystem:
> Client pengine already running as pid 4243
> Jan 17 12:00:14 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_takeover:
> Taking over DC status for this partition
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/O mode
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/117, version=1.8.26): ok (rc=0)
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/W mode
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/118,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/119,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/121, version=1.8.26): ok (rc=0)
> Jan 17 12:00:14 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_join_offer_all:
> join-6: Waiting on 2 outstanding join acks
> Jan 17 12:00:14 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/123, version=1.8.26): ok (rc=0)
> Jan 17 12:00:14 vsa-0000003ca-vc-0 crmd: [1120]: info:
> config_query_callback: Checking for expired actions every 900000ms
> Jan 17 12:00:15 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 8 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:15 vsa-0000003ca-vc-0 crmd: [1120]: info: update_dc: Set DC to
> vsa-0000003ca-vc-0 (3.0.1)
> Jan 17 12:00:15 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:15 vsa-0000003ca-vc-0 crmd: [1120]: info: update_dc: Unset DC
> vsa-0000003ca-vc-0
> Jan 17 12:00:15 vsa-0000003ca-vc-0 crmd: [1120]: WARN: do_log: FSA: Input
> I_JOIN_REQUEST from route_message() received in state S_ELECTION
> Jan 17 12:00:16 vsa-0000003ca-vc-0 crmd: [1120]: info:
> do_election_count_vote: Election 9 (owner:
> 00000970-0970-0001-2b91-000000000001) pass: vote from vsa-0000003ca-vc-1
> (Age)
> Jan 17 12:00:17 vsa-0000003ca-vc-0 crmd: [1120]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Jan 17 12:00:17 vsa-0000003ca-vc-0 crmd: [1120]: info: start_subsystem:
> Starting sub-system "pengine"
> Jan 17 12:00:17 vsa-0000003ca-vc-0 crmd: [1120]: WARN: start_subsystem:
> Client pengine already running as pid 4243
> Jan 17 12:00:21 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_takeover:
> Taking over DC status for this partition
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/O mode
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/126, version=1.8.26): ok (rc=0)
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_readwrite:
> We are now in R/W mode
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/127,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/128,
> version=1.8.26): ok (rc=0)
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/130, version=1.8.26): ok (rc=0)
> Jan 17 12:00:21 vsa-0000003ca-vc-0 crmd: [1120]: info: do_dc_join_offer_all:
> join-7: Waiting on 2 outstanding join acks
> Jan 17 12:00:21 vsa-0000003ca-vc-0 cib: [1116]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/132, version=1.8.26): ok (rc=0)
> Jan 17 12:00:21 vsa-0000003ca-vc-0 crmd: [1120]: info:
> config_query_callback: Checking for expired actions every 900000ms
>
>
> On node 2:
> Jan 17 12:00:02 vsa-0000003ca-vc-1 crmd: [1104]: info: crm_timer_popped:
> Election Trigger (I_DC_TIMEOUT) just popped!
> Jan 17 12:00:02 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Jan 17 12:00:02 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Jan 17 12:00:03 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:03 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:03 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_ELECTION -> S_PENDING [ input=I_PENDING
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:03 vsa-0000003ca-vc-1 crmd: [1104]: info: do_dc_release: DC
> role released
> Jan 17 12:00:03 vsa-0000003ca-vc-1 crmd: [1104]: info: do_te_control:
> Transitioner is now inactive
> Jan 17 12:00:09 vsa-0000003ca-vc-1 crmd: [1104]: info: crm_timer_popped:
> Election Trigger (I_DC_TIMEOUT) just popped!
> Jan 17 12:00:09 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Jan 17 12:00:09 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Jan 17 12:00:09 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:10 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_ELECTION -> S_PENDING [ input=I_PENDING
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:10 vsa-0000003ca-vc-1 crmd: [1104]: info: do_dc_release: DC
> role released
> Jan 17 12:00:10 vsa-0000003ca-vc-1 crmd: [1104]: info: do_te_control:
> Transitioner is now inactive
> Jan 17 12:00:15 vsa-0000003ca-vc-1 crmd: [1104]: info: crm_timer_popped:
> Election Trigger (I_DC_TIMEOUT) just popped!
> Jan 17 12:00:15 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Jan 17 12:00:15 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Jan 17 12:00:16 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:16 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_ELECTION -> S_PENDING [ input=I_PENDING
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:16 vsa-0000003ca-vc-1 crmd: [1104]: info: do_dc_release: DC
> role released
> Jan 17 12:00:16 vsa-0000003ca-vc-1 crmd: [1104]: info: do_te_control:
> Transitioner is now inactive
> Jan 17 12:00:22 vsa-0000003ca-vc-1 crmd: [1104]: info: crm_timer_popped:
> Election Trigger (I_DC_TIMEOUT) just popped!
> Jan 17 12:00:22 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Jan 17 12:00:22 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Jan 17 12:00:22 vsa-0000003ca-vc-1 crmd: [1104]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Jan 17 12:00:23 vsa-0000003ca-vc-1 crmd: [1104]: info: do_state_transition:
> State transition S_ELECTION -> S_PENDING [ input=I_PENDING
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 17 12:00:23 vsa-0000003ca-vc-1 crmd: [1104]: info: do_dc_release: DC
> role released
> Jan 17 12:00:23 vsa-0000003ca-vc-1 crmd: [1104]: info: do_te_control:
> Transitioner is now inactive
>
> --Shyam
>
> On Thu, Feb 2, 2012 at 3:28 PM, Andrew Beekhof <andrew at beekhof.net> wrote:
>>
>> They both think they should be the DC.
>> But the log fragments dont extend back far enough to say why.
>>
>> On Fri, Jan 27, 2012 at 10:21 PM, Shyam <shyam.kaushik at gmail.com> wrote:
>> > Folks,
>> >
>> > We are constantly running into a long election cycle where in a 2-node
>> > cluster when both of them are simultaneously rebooted, they take a long
>> > time
>> > running through election loop.
>> >
>> > On one node pacemaker loops like:
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info: do_dc_takeover:
>> > Taking
>> > over DC status for this partition
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_readwrite:
>> > We are now in R/O mode
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_slave_all for section 'all'
>> > (origin=local/crmd/222, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_readwrite:
>> > We are now in R/W mode
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_master for section 'all'
>> > (origin=local/crmd/223,
>> > version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section cib
>> > (origin=local/crmd/224,
>> > version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section crm_config
>> > (origin=local/crmd/226, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_dc_join_offer_all:
>> > join-25: Waiting on 2 outstanding join acks
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section crm_config
>> > (origin=local/crmd/228, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > config_query_callback:
>> > Checking for expired actions every 900000ms
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_election_count_vote: Election 50 (owner:
>> > 00000156-0156-0000-2b91-000000000000) pass: vote from vsa-0000009c-vc-0
>> > (Age)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info: update_dc: Set DC
>> > to
>> > vsa-0000009c-vc-1 (3.0.1)
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_state_transition:
>> > State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
>> > cause=C_FSA_INTERNAL origin=do_election_count_vote ]
>> > Jan 26 22:03:20 vsa-0000009c-vc-1 crmd: [1134]: info: update_dc: Unset
>> > DC
>> > vsa-0000009c-vc-1
>> > Jan 26 22:03:21 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_election_count_vote: Election 51 (owner:
>> > 00000156-0156-0000-2b91-000000000000) pass: vote from vsa-0000009c-vc-0
>> > (Age)
>> > Jan 26 22:03:21 vsa-0000009c-vc-1 crmd: [1134]: WARN: do_log: FSA: Input
>> > I_JOIN_REQUEST from route_message() received in state S_ELECTION
>> > Jan 26 22:03:22 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_state_transition:
>> > State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
>> > cause=C_FSA_INTERNAL origin=do_election_check ]
>> > Jan 26 22:03:22 vsa-0000009c-vc-1 crmd: [1134]: info: start_subsystem:
>> > Starting sub-system "pengine"
>> > Jan 26 22:03:22 vsa-0000009c-vc-1 crmd: [1134]: WARN: start_subsystem:
>> > Client pengine already running as pid 1234
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info: do_dc_takeover:
>> > Taking
>> > over DC status for this partition
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_readwrite:
>> > We are now in R/O mode
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_slave_all for section 'all'
>> > (origin=local/crmd/231, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_readwrite:
>> > We are now in R/W mode
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_master for section 'all'
>> > (origin=local/crmd/232,
>> > version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section cib
>> > (origin=local/crmd/233,
>> > version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section crm_config
>> > (origin=local/crmd/235, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_dc_join_offer_all:
>> > join-26: Waiting on 2 outstanding join acks
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 cib: [1130]: info:
>> > cib_process_request:
>> > Operation complete: op cib_modify for section crm_config
>> > (origin=local/crmd/237, version=1.1.1): ok (rc=0)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > config_query_callback:
>> > Checking for expired actions every 900000ms
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_election_count_vote: Election 52 (owner:
>> > 00000156-0156-0000-2b91-000000000000) pass: vote from vsa-0000009c-vc-0
>> > (Age)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info: update_dc: Set DC
>> > to
>> > vsa-0000009c-vc-1 (3.0.1)
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_state_transition:
>> > State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
>> > cause=C_FSA_INTERNAL origin=do_election_count_vote ]
>> > Jan 26 22:03:26 vsa-0000009c-vc-1 crmd: [1134]: info: update_dc: Unset
>> > DC
>> > vsa-0000009c-vc-1
>> > Jan 26 22:03:27 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_election_count_vote: Election 53 (owner:
>> > 00000156-0156-0000-2b91-000000000000) pass: vote from vsa-0000009c-vc-0
>> > (Age)
>> > Jan 26 22:03:27 vsa-0000009c-vc-1 crmd: [1134]: WARN: do_log: FSA: Input
>> > I_JOIN_REQUEST from route_message() received in state S_ELECTION
>> > Jan 26 22:03:28 vsa-0000009c-vc-1 crmd: [1134]: info:
>> > do_state_transition:
>> > State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
>> > cause=C_FSA_INTERNAL origin=do_election_check ]
>> > Jan 26 22:03:28 vsa-0000009c-vc-1 crmd: [1134]: info: start_subsystem:
>> > Starting sub-system "pengine"
>> > Jan 26 22:03:28 vsa-0000009c-vc-1 crmd: [1134]: WARN: start_subsystem:
>> > Client pengine already running as pid 1234
>> >
>> > &  other node with
>> > Jan 26 22:03:20 vsa-0000009c-vc-0 crmd: [1314]: info: crm_timer_popped:
>> > Election Trigger (I_DC_TIMEOUT) just popped!
>> > Jan 26 22:03:20 vsa-0000009c-vc-0 crmd: [1314]: WARN: do_log: FSA: Input
>> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
>> > Jan 26 22:03:20 vsa-0000009c-vc-0 crmd: [1314]: info:
>> > do_state_transition:
>> > State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
>> > cause=C_TIMER_POPPED origin=crm_timer_popped ]
>> > Jan 26 22:03:21 vsa-0000009c-vc-0 crmd: [1314]: WARN: do_log: FSA: Input
>> > I_JOIN_OFFER from route_message() received in state S_ELECTION
>> > Jan 26 22:03:21 vsa-0000009c-vc-0 crmd: [1314]: info:
>> > do_state_transition:
>> > State transition S_ELECTION -> S_PENDING [ input=I_PENDING
>> > cause=C_FSA_INTERNAL origin=do_election_count_vote ]
>> > Jan 26 22:03:21 vsa-0000009c-vc-0 crmd: [1314]: info: do_dc_release: DC
>> > role
>> > released
>> > Jan 26 22:03:21 vsa-0000009c-vc-0 crmd: [1314]: info: do_te_control:
>> > Transitioner is now inactive
>> > Jan 26 22:03:26 vsa-0000009c-vc-0 crmd: [1314]: info: crm_timer_popped:
>> > Election Trigger (I_DC_TIMEOUT) just popped!
>> > Jan 26 22:03:26 vsa-0000009c-vc-0 crmd: [1314]: WARN: do_log: FSA: Input
>> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
>> > Jan 26 22:03:26 vsa-0000009c-vc-0 crmd: [1314]: info:
>> > do_state_transition:
>> > State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
>> > cause=C_TIMER_POPPED origin=crm_timer_popped ]
>> > Jan 26 22:03:27 vsa-0000009c-vc-0 crmd: [1314]: WARN: do_log: FSA: Input
>> > I_JOIN_OFFER from route_message() received in state S_ELECTION
>> > Jan 26 22:03:27 vsa-0000009c-vc-0 crmd: [1314]: info:
>> > do_state_transition:
>> > State transition S_ELECTION -> S_PENDING [ input=I_PENDING
>> > cause=C_FSA_INTERNAL origin=do_election_count_vote ]
>> > Jan 26 22:03:27 vsa-0000009c-vc-0 crmd: [1314]: info: do_dc_release: DC
>> > role
>> > released
>> > Jan 26 22:03:27 vsa-0000009c-vc-0 crmd: [1314]: info: do_te_control:
>> > Transitioner is now inactive
>> >
>> > This takes several minutes & finally breaks.
>> >
>> > Any pointers on what can be causing this?
>> >
>> > Thanks.
>> >
>> > --Shyam
>> >
>> > _______________________________________________
>> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >
>> > Project Home: http://www.clusterlabs.org
>> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> > Bugs: http://bugs.clusterlabs.org
>> >
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>