[Pacemaker] solaris problem
Andrei Belov
defanator at gmail.com
Mon Mar 25 12:01:58 UTC 2013
I've rebuilt libqb using separated SOCKETDIR (/var/run/qb), and set hacluster:haclient ownership to this dir.
After that pacemakerd has been successfully started with all its childs:
[root at ha1 /var/run/qb]# pacemakerd -fV
Could not establish pacemakerd connection: Connection refused (146)
info: crm_ipc_connect: Could not establish pacemakerd connection: Connection refused (146)
info: get_cluster_type: Detected an active 'corosync' cluster
info: read_config: Reading configure for stack: corosync
notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
notice: main: Starting Pacemaker 1.1.8 (Build: 1f8858c): ncurses libqb-logging libqb-ipc upstart systemd corosync-native
info: main: Maximum core file size is: 18446744073709551613
info: qb_ipcs_us_publish: server name: pacemakerd
notice: update_node_processes: 48de70 Node 182452614 now known as ha1, was:
info: start_child: Forked child 60719 for process cib
info: start_child: Forked child 60720 for process stonith-ng
info: start_child: Forked child 60721 for process lrmd
info: start_child: Forked child 60722 for process attrd
info: start_child: Forked child 60723 for process pengine
info: start_child: Forked child 60724 for process crmd
info: main: Starting mainloop
[root at ha1 /var/run/qb]# ls -l
total 0
srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 attrd
srwxrwxrwx 1 root root 0 Mar 25 11:43 cfg
srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_ro
srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_rw
srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 cib_shm
srwxrwxrwx 1 root root 0 Mar 25 11:43 cmap
srwxrwxrwx 1 root root 0 Mar 25 11:43 cpg
srwxrwxrwx 1 root root 0 Mar 25 11:50 lrmd
srwxrwxrwx 1 root root 0 Mar 25 11:50 pacemakerd
srwxrwxrwx 1 hacluster root 0 Mar 25 11:50 pengine
srwxrwxrwx 1 root root 0 Mar 25 11:43 quorum
srwxrwxrwx 1 root root 0 Mar 25 11:50 stonith-ng
However, libqb still can not create some files in /var/run due to insufficient permissions:
Mar 25 11:50:45 [60719] cib: info: init_cs_connection_once: Connection to 'corosync': established
Mar 25 11:50:45 [60719] cib: info: crm_get_peer: Node 182452614 is now known as ha1
Mar 25 11:50:45 [60719] cib: info: crm_get_peer: Node 182452614 has uuid 182452614
Mar 25 11:50:45 [60719] cib: info: qb_ipcs_us_publish: server name: cib_ro
Mar 25 11:50:45 [60719] cib: info: qb_ipcs_us_publish: server name: cib_rw
Mar 25 11:50:45 [60719] cib: info: qb_ipcs_us_publish: server name: cib_shm
Mar 25 11:50:45 [60719] cib: info: cib_init: Starting cib mainloop
Mar 25 11:50:45 [60719] cib: info: pcmk_cpg_membership: Joined[0.0] cib.182452614
Mar 25 11:50:45 [60719] cib: info: pcmk_cpg_membership: Member[0.0] cib.182452614
Mar 25 11:50:45 [60719] cib: info: pcmk_cpg_membership: Member[0.1] cib.182452614
Mar 25 11:50:46 [60719] cib: error: qb_sys_mmap_file_open: couldn't open file /var/run/qb-cib_rw-control-60719-60720-15: Permission denied (13)
Mar 25 11:50:46 [60719] cib: error: qb_ipcs_us_connect: couldn't create file for mmap (60719-60720-15): Permission denied (13)
Mar 25 11:50:46 [60719] cib: error: handle_new_connection: Invalid IPC credentials (60719-60720-15).
Mar 25 11:50:46 [60720] stonith-ng: info: crm_ipc_connect: Could not establish cib_rw connection: Permission denied (13)
Mar 25 11:50:46 [60719] cib: error: qb_sys_mmap_file_open: couldn't open file /var/run/qb-cib_shm-control-60719-60724-16: Permission denied (13)
Mar 25 11:50:46 [60719] cib: error: qb_ipcs_us_connect: couldn't create file for mmap (60719-60724-16): Permission denied (13)
Mar 25 11:50:46 [60719] cib: error: handle_new_connection: Invalid IPC credentials (60719-60724-16).
Mar 25 11:50:46 [60724] crmd: info: crm_ipc_connect: Could not establish cib_shm connection: Permission denied (13)
Mar 25 11:50:46 [60724] crmd: info: do_cib_control: Could not connect to the CIB service: Transport endpoint is not connected
Mar 25 11:50:46 [60724] crmd: warning: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry
If someone has working setup on Linux with corosync 2.x, libqb and pacemaker 1.1.x - I'd be very appreciated for sharing some information about a places which libqb uses for its special socket files.
Thanks in advance!
(Can we say now that this problem is libqb-related, not pacemaker?)
On Mar 25, 2013, at 15:30 , Andrei Belov <defanator at gmail.com> wrote:
> Andreas,
>
> just tried "PCMK_ipc_type=socket pacemaker -fV" - a bunch of additional "event_send" errors appeared:
>
> Mar 25 11:15:55 [33641] ha1 corosync error [MAIN ] event_send retuned -32, expected 256!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 217!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 219!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 256!
> Mar 25 11:15:55 [53980] pengine: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/pengine): Permission denied (13)
> Mar 25 11:15:55 [53980] pengine: error: mainloop_add_ipc_server: Could not start pengine IPC server: Unknown error (-13)
> Mar 25 11:15:55 [53980] pengine: error: main: Couldn't start IPC server
> Mar 25 11:15:55 [53975] pacemakerd: error: pcmk_child_exit: Child process pengine exited (pid=53980, rc=1)
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 256!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [53979] attrd: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/attrd): Permission denied (13)
> Mar 25 11:15:55 [53979] attrd: error: mainloop_add_ipc_server: Could not start attrd IPC server: Unknown error (-13)
> Mar 25 11:15:55 [53979] attrd: error: main: Could not start IPC server
> Mar 25 11:15:55 [53979] attrd: error: main: Aborting startup
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [53975] pacemakerd: error: pcmk_child_exit: Child process attrd exited (pid=53979, rc=100)
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 256!
> Mar 25 11:15:55 [53976] cib: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/cib_ro): Permission denied (13)
> Mar 25 11:15:55 [53976] cib: error: mainloop_add_ipc_server: Could not start cib_ro IPC server: Unknown error (-13)
> Mar 25 11:15:55 [53976] cib: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/cib_rw): Permission denied (13)
> Mar 25 11:15:55 [53976] cib: error: mainloop_add_ipc_server: Could not start cib_rw IPC server: Unknown error (-13)
> Mar 25 11:15:55 [53976] cib: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/cib_shm): Permission denied (13)
> Mar 25 11:15:55 [53976] cib: error: mainloop_add_ipc_server: Could not start cib_shm IPC server: Unknown error (-13)
> Mar 25 11:15:55 [53976] cib: error: cib_init: Couldnt start all IPC channels, exiting.
> Mar 25 11:15:55 [53975] pacemakerd: error: pcmk_child_exit: Child process cib exited (pid=53976, rc=255)
> Mar 25 11:15:55 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 223!
> Mar 25 11:16:04 [53977] stonith-ng: error: setup_cib: Could not connect to the CIB service: -134 fffffd7fc421a0b0
> Mar 25 11:16:04 [33641] ha1 corosync error [SERV ] event_send retuned -32, expected 217!
> Mar 25 11:16:04 [53975] pacemakerd: notice: pcmk_shutdown_worker: Attempting to inhibit respawning after fatal error
>
>
> # fgrep 32 /usr/include/sys/errno.h
> #define EPIPE 32 /* Broken pipe */
>
>
>
> On Mar 25, 2013, at 13:55 , "Grüninger, Andreas (LGL Extern)" <Andreas.Grueninger at lgl.bwl.de> wrote:
>
>> With solaris/openindiana you should use this setting
>> export PCMK_ipc_type=socket
>>
>> Andreas
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Andrei Belov [mailto:defanator at gmail.com]
>> Gesendet: Montag, 25. März 2013 10:43
>> An: pacemaker at oss.clusterlabs.org
>> Betreff: [Pacemaker] solaris problem
>>
>> Hi folks,
>>
>> I'm trying to build test HA cluster on Solaris 5.11 using libqb 0.14.4, corosync 2.3.0 and pacemaker 1.1.8, and I'm facing a strange problem while starting pacemaker.
>>
>> Log shows the following errors:
>>
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: mainloop_add_ipc_server: Could not start lrmd IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33720] lrmd: error: try_server_create: New IPC server could not be created because another lrmd process exists, sending shutdown command to old lrmd process.
>> Mar 25 09:21:26 [33720] lrmd: error: main: Failed to allocate lrmd server. shutting down
>> Mar 25 09:21:26 [33722] pengine: error: mainloop_add_ipc_server: Could not start pengine IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33722] pengine: error: main: Couldn't start IPC server
>> Mar 25 09:21:26 [33717] pacemakerd: error: pcmk_child_exit: Child process lrmd exited (pid=33720, rc=255)
>> Mar 25 09:21:26 [33721] attrd: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/attrd): Permission denied (13)
>> Mar 25 09:21:26 [33721] attrd: error: mainloop_add_ipc_server: Could not start attrd IPC server: Unknown error (-13)
>> Mar 25 09:21:26 [33721] attrd: error: main: Could not start IPC server
>> Mar 25 09:21:26 [33721] attrd: error: main: Aborting startup
>> Mar 25 09:21:26 [33717] pacemakerd: error: pcmk_child_exit: Child process pengine exited (pid=33722, rc=1)
>> Mar 25 09:21:26 [33717] pacemakerd: error: pcmk_child_exit: Child process attrd exited (pid=33721, rc=100)
>> Mar 25 09:21:26 [33718] cib: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/cib_ro): Permission denied (13)
>> Mar 25 09:21:26 [33718] cib: error: mainloop_add_ipc_server: Could not start cib_ro IPC server: Unknown error (-13)
>> Mar 25 09:21:26 [33718] cib: error: qb_ipcs_us_publish: Could not bind AF_UNIX (/var/run/cib_rw): Permission denied (13)
>> Mar 25 09:21:26 [33718] cib: error: mainloop_add_ipc_server: Could not start cib_rw IPC server: Unknown error (-13)
>> Mar 25 09:21:26 [33718] cib: error: mainloop_add_ipc_server: Could not start cib_shm IPC server: Unknown error (-48)
>> Mar 25 09:21:26 [33718] cib: error: cib_init: Couldnt start all IPC channels, exiting.
>> Mar 25 09:21:26 [33717] pacemakerd: error: pcmk_child_exit: Child process cib exited (pid=33718, rc=255)
>> Mar 25 09:21:35 [33719] stonith-ng: error: setup_cib: Could not connect to the CIB service: -134 fffffd7fc421a0b0
>> Mar 25 09:21:35 [33717] pacemakerd: notice: pcmk_shutdown_worker: Attempting to inhibit respawning after fatal error
>>
>> Full log (in case of any things I've probably missed) is attached.
>>
>> I wonder to know the reason of "unknown error (-48)" - on this system 48 in errno.h is "ENOTSUP", but I haven't found the exact place in code where this may happen (so I'm not sure about that).
>>
>> Just for record - I'm able to run corosync on two nodes and see them connected without any visible problems - thus, I suppose there may be something wrong with either pacemaker or libqb.
>>
>> Any help will be greatly appreciated!
>>
>> Thanks,
>> Andrei.
>>
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
More information about the Pacemaker
mailing list