<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi everyone.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have 2 nodes running on ESX hosts in 2 geographically diverse data centres. The link between them is a DWDM fibre link which is the only thing I can think of as being the cause of this.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>SLES 11 SP1 with HAE. All latest updates.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If Corosync is set to Multicast on the default address, there are no comms between Corosync on the nodes. If I use broadcast, it will communicate and let the nodes join.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If I reboot node 2, it rejoins fine. If I reboot node 1, it enters a pending phase for a while then just drops to offline. I can then clear the config out again and let the nodes rejoin. Node 1 always seems to be the DC.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Pending – logs from node 1, loops this every second:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>-02: id=336371722 state=member (new) addr=r(0) ip(10.160.12.20) votes=1 born=7912 seen=7920 proc=00000000000000000000000000151312<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: crm_update_peer: Node PPS-VMAIL-01: id=168599562 state=member (new) addr=r(0) ip(10.160.12.10) (new) votes=1 (new) born=7920 seen=7920 proc=00000000000000000000000000151312 (new)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: WARN: do_log: FSA: Input I_SHUTDOWN from revision_check_callback() received in state S_STARTING<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_state_transition: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_FSA_INTERNAL origin=revision_check_callback ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_lrm_control: Disconnected from the LRM<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_ha_control: Disconnected from OpenAIS<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_cib_control: Disconnecting CIB<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: free_mem: Dropping I_NULL: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=register_fsa_error_adv ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:37:13 PPS-VMAIL-01 crmd: [3896]: info: do_exit: [crmd] stopped (0)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Offline – logs from node 1, loops every second:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 cib: [3510]: info: cib_replace_notify: Local-only Replace: 0.0.0 from PP2-VMAIL-02<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 attrd: [3512]: info: do_cib_replaced: Sending full refresh<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 attrd: [3512]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (<null>)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 cib: [3510]: info: apply_xml_diff: Digest mis-match: expected 0cf389141d344ca552679f9924d281c5, calculated 818a100a0e3b725068393624381c9d4f<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 cib: [3510]: notice: cib_process_diff: Diff 0.13.642 -> 0.0.0 not applied to 0.13.642: Failed application of an update diff<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 cib: [3510]: info: cib_server_process_diff: Requesting re-sync from peer<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:38:06 PPS-VMAIL-01 cib: [3510]: WARN: cib_diff_notify: Local-only Change (client:attrd, call: 1221): 0.0.0 (Application of an update diff failed, requesting a full refresh)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Offline – logs from node 2, loops every second:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:39:05 PP2-VMAIL-02 corosync[3794]: [TOTEM ] Retransmit List: 29b7 29b8 29b9<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:39:05 PP2-VMAIL-02 corosync[3794]: [TOTEM ] Retransmit List: 29bb 29bc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Courier New"'>Apr 2 14:39:05 PP2-VMAIL-02 cib: [3801]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=PPS-VMAIL-01/PPS-VMAIL-01/(null), version=0.13.1538): ok (rc=0)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any ideas please?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#00527F;mso-fareast-language:EN-GB'>Darren Mansell</span></b><span style='font-size:12.0pt;font-family:"Times New Roman","serif";color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>