[Pacemaker] communications problems in cluster

Саша Александров shurrman at gmail.com
Mon Oct 13 14:31:38 UTC 2014 

Hi!

Most likely related...
I have node vm-vmwww with remote-node vmwww. Both are reported online
(vmwww:vm-vmwww) and vm-vmwww is reported as 'started on wings1'.
However, when I try to cleanup faulty failed action " vmwww_start_0 on
wings1 'unknown error' (1): call=100, status=Timed Out ", here is what I
get in the log:

Oct 13 18:25:43 wings1 crmd[3844]:  warning: qb_ipcs_event_sendv:
new_event_notification (3844-18918-16): Broken pipe (32)
Oct 13 18:25:43 wings1 crmd[3844]:    error: do_lrm_invoke: no lrmd
connection for remote node vmwww found on cluster node wings1. Can not
process request.
Oct 13 18:25:43 wings1 crmd[3844]:    error: send_msg_via_ipc: Unknown
Sub-system (d483a600-5535-4f0d-8ffd-2af391f5cb21)... discarding message.
Oct 13 18:25:43 wings1 crmd[3844]:    error: send_msg_via_ipc: Unknown
Sub-system (d483a600-5535-4f0d-8ffd-2af391f5cb21)... discarding message.
Oct 13 18:25:43 wings1 crmd[3844]:    error: send_msg_via_ipc: Unknown
Sub-system (d483a600-5535-4f0d-8ffd-2af391f5cb21)... discarding message.
Oct 13 18:25:43 wings1 crmd[3844]:    error: send_msg_via_ipc: Unknown
Sub-system (d483a600-5535-4f0d-8ffd-2af391f5cb21)... discarding message.

I go to the VM, and try to run 'crm_mon':

Oct 13 18:27:06 vmwww pacemaker_remoted[3798]:    error: ipc_proxy_accept:
No ipc providers available for uid 0 gid 0
Oct 13 18:27:06 vmwww pacemaker_remoted[3798]:    error:
handle_new_connection: Error in connection setup (3798-3868-13): Remote I/O
error (121)

ps aux | grep pace
root      3798  0.1  0.1  76396  2868 ?        S    18:16   0:00
pacemaker_remoted

netstat -nltp | grep 3121
tcp        0      0 0.0.0.0:3121                0.0.0.0:*
LISTEN      3798/pacemaker_remo

However I can telnet ok:

[root at wings1 ~]# telnet vmwww 3121
Trying 192.168.222.89...
Connected to vmwww.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

This is pretty weird...

Best regards,
Alex


2014-10-13 17:47 GMT+04:00 Саша Александров <shurrman at gmail.com>:

> Hi!
>
> I was building a cluster with pacemaker+pacemaker-remote  (CentOS 6.5,
> everything from the official repo).
> While I had several resources, everything was fine. However, when I added
> more VMs (2 nodes and 10 VMs currently) I started to run into problems (see
> below).
> Strange thing is that when I start cman/pacemaker some time later - they
> seem to work fine for some time.
>
> Oct 13 17:03:54 wings1 pacemakerd[26440]:   notice: pcmk_child_exit: Child
> process crmd terminated with signal 13 (pid=30010, core=0)
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: qb_ipcs_event_sendv:
> new_event_notification (26448-30010-6): Bad file descriptor (9)
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
> Oct 13 17:03:54 wings1 pacemakerd[26440]:   notice: pcmk_process_exit:
> Respawning failed child process: crmd
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
> Oct 13 17:03:54 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/665bd130-2630-454b-9102-3f17d2bd71f3 failed
>
> Oct 13 17:03:57 wings1 pacemakerd[26440]:   notice: pcmk_child_exit: Child
> process crmd terminated with signal 13 (pid=30603, core=0)
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: qb_ipcs_event_sendv:
> new_event_notification (26448-30603-6): Bad file descriptor (9)
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 pacemakerd[26440]:   notice: pcmk_process_exit:
> Respawning failed child process: crmd
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/820ac884-24ca-4fff-9dc8-0a09e82e0e0a failed
> Oct 13 17:03:57 wings1 crmd[31192]:   notice: crm_add_logfile: Additional
> logging available in /var/log/cluster/corosync.log
> Oct 13 17:03:57 wings1 cib[26446]:  warning: qb_ipcs_event_sendv:
> new_event_notification (26446-30603-11): Broken pipe (32)
> Oct 13 17:03:57 wings1 cib[26446]:  warning: cib_notify_send_one:
> Notification of client crmd/fe944296-b3a1-4177-a94c-650568e8ff0a failed
>
> ..................
>
> So it keeps restarting, I even had to unmanage resources and stop
> pacemaker/cman.
>
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: qb_ipcs_event_sendv:
> new_event_notification (26448-32444-6): Bad file descriptor (9)
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 pacemakerd[26440]:   notice: pcmk_child_exit: Child
> process crmd terminated with signal 13 (pid=32444, core=0)
> Oct 13 17:04:13 wings1 pacemakerd[26440]:   notice: pcmk_process_exit:
> Respawning failed child process: crmd
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 lrmd[26448]:  warning: send_client_notify:
> Notification of client crmd/ea7ab099-1005-450b-9e46-d9d13ea266e4 failed
> Oct 13 17:04:13 wings1 cib[26446]:  warning: qb_ipcs_event_sendv:
> new_event_notification (26446-32444-11): Broken pipe (32)
> Oct 13 17:04:13 wings1 cib[26446]:  warning: cib_notify_send_one:
> Notification of client crmd/ef727424-ce2b-4b3b-8749-82136dc72af8 failed
>
>
>
> And one more thing (probably not related, but who knows) - I have CentOS
> 7.0 on one of the VMs, LRMD is unable to establish communications with
> pacemaker_remote on that VM:
>
> (node):
> Oct 13 17:31:43 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 6.
> Oct 13 17:31:45 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 7.
> Oct 13 17:31:47 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 8.
> Oct 13 17:31:48 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 9.
> Oct 13 17:31:50 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 10.
> Oct 13 17:31:51 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 11.
> Oct 13 17:31:53 wings1 crmd[3844]:    error: lrmd_tls_send_recv: Remote
> lrmd server disconnected while waiting for reply with id 12.
>
> (VM):
> Oct 13 21:27:32 bank systemd: Started Pacemaker Remote Service.
> Oct 13 21:27:32 bank pacemaker_remoted: Cannot change active directory to
> /var/lib/pacemaker/cores: No such file or directory (2)
> Oct 13 21:27:32 bank pacemaker_remoted[1853]: notice:
> lrmd_init_remote_tls_server: Starting a tls listener on port 3121.
> Oct 13 21:27:32 bank pacemaker_remoted[1853]: notice: bind_and_listen:
> Listening on address ::
> Oct 13 21:31:39 bank pacemaker_remoted[1853]: notice: lrmd_remote_listen:
> LRMD client connection established. 0x1c49d60 id:
> de49ea57-e94c-45bf-9d2d-d0f36cb2c4f7
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_abort:
> crm_remote_header: Triggered assert at remote.c:118 : endian == ENDIAN_LOCAL
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_remote_header:
> Invalid message detected, endian mismatch: badadbbd is neither 6d726c3c nor
> the swab'd 3c6
> c726d
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_abort:
> crm_remote_header: Triggered assert at remote.c:118 : endian == ENDIAN_LOCAL
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_remote_header:
> Invalid message detected, endian mismatch: badadbbd is neither 6d726c3c nor
> the swab'd 3c6c726d
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_abort:
> crm_remote_header: Triggered assert at remote.c:118 : endian == ENDIAN_LOCAL
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: error: crm_remote_header:
> Invalid message detected, endian mismatch: badadbbd is neither 6d726c3c nor
> the swab'd 3c6c726d
> Oct 13 21:31:40 bank pacemaker_remoted[1853]: notice:
> lrmd_remote_client_destroy: LRMD client disconnecting remote client - name:
> <unknown> id: de49ea57-e94c-45bf-9d2d-d0f36cb2c4f7
>
>
>
> --
> Best regards,
> Alexandr
>



-- 
С уважением, ААА.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20141013/98b2e55a/attachment.htm>

More information about the Pacemaker mailing list