[Pacemaker] cibadmin -Q: Call cib_query failed (-62): Timer expired

[Radoslaw Garbacz]

Hi,

I have a problem starting up a cluster after upgrading corosync from
1.4 to 2.3.2 and pacemaker from 1.8 to 1.9.

All "crm_node" calls report well, but any CIB manipulation fails, i.e.:
* crm_node -q: 1
* crm_node -l: OK
* crm_node -p: OK
* cibadmin -Q: Call cib_query failed (-62): Timer expired

No iptables, no SELinux, 3 nodes cluster, corosync.conf:
...
        ringnumber: 0
        bindnetaddr: ...
        mcastport: 7800
    }

    transport: udpu



Any help greatly appreciated.


Below is some more information:

* pacemaker logs:

Sep 26 22:24:00 [2836] ip-10-114-210-162        cib:     info:
crm_client_new:  Connecting 0x111b780 for uid=0 gid=0 pid=2883
id=977d6f23-963b-41a4-8fe0-a63024080d41
Sep 26 22:24:00 [2836] ip-10-114-210-162        cib:     info:
cib_process_request:     Forwarding cib_query operation for section
'all' to master (origin=local/cibadmin/2)
Sep 26 22:24:30 [2836] ip-10-114-210-162        cib:     info:
crm_client_destroy:      Destroying 0 events


* ps axf | grep pacemaker|corosync:

2806 ?        Ssl    0:10 corosync
 2834 pts/1    S      0:00 pacemakerd
 2836 ?        Ss     0:01  \_ /usr/libexec/pacemaker/cib
 2837 ?        Ss     0:00  \_ /usr/libexec/pacemaker/stonithd
 2838 ?        Ss     0:00  \_ /usr/libexec/pacemaker/lrmd
 2839 ?        Ss     0:00  \_ /usr/libexec/pacemaker/attrd
 2840 ?        Ss     0:00  \_ /usr/libexec/pacemaker/pengine
 2841 ?        Ss     0:00  \_ /usr/libexec/pacemaker/crmd


* strace cibadmin -Q:

open("/dev/shm/qb-cib_rw-event-2836-2897-12-data", O_RDWR) = 6
ftruncate(6, 20480000)                  = 0
mmap(NULL, 40960000, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fa221692000
mmap(0x7fa221692000, 20480000, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_FIXED, 6, 0) = 0x7fa221692000
mmap(0x7fa222a1a000, 20480000, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_FIXED, 6, 0) = 0x7fa222a1a000
close(6)                                = 0
close(5)                                = 0
close(6)                                = -1 EBADF (Bad file descriptor)
fstat(4, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
fcntl(4, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)
sendto(4, "~", 1, MSG_NOSIGNAL, NULL, 0) = 1
futex(0x7fa22df4cb60, FUTEX_WAKE_PRIVATE, 2147483647) = 0
gettimeofday({1380234692, 68879}, NULL) = 0
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)
gettimeofday({1380234692, 69522}, NULL) = 0
sendto(4, "\274", 1, MSG_NOSIGNAL, NULL, 0) = 1
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)
gettimeofday({1380234692, 70085}, NULL) = 0
gettimeofday({1380234692, 70197}, NULL) = 0
poll([{fd=4, events=POLLIN}], 1, 30000) = 0 (Timeout)
gettimeofday({1380234722, 91625}, NULL) = 0
write(2, "Call cib_query failed (-62): Tim"..., 43Call cib_query
failed (-62): Timer expired
) = 43
poll([{fd=4, events=POLLIN}], 1, 0)     = 0 (Timeout)


* netstat -lxp:

Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program
name    Path
unix  2      [ ACC ]     STREAM     LISTENING     20021  2836/cib
      @cib_rw
unix  2      [ ACC ]     STREAM     LISTENING     19958  2838/lrmd
      @lrmd
unix  2      [ ACC ]     STREAM     LISTENING     19789  2806/corosync
      @quorum
unix  2      [ ACC ]     STREAM     LISTENING     19786  2806/corosync
      @cmap
unix  2      [ ACC ]     STREAM     LISTENING     20020  2836/cib
      @cib_ro
unix  2      [ ACC ]     STREAM     LISTENING     20057  2837/stonithd
      @stonith-ng
unix  2      [ ACC ]     STREAM     LISTENING     19787  2806/corosync
      @cfg
unix  2      [ ACC ]     STREAM     LISTENING     19906
2834/pacemakerd     @pacemakerd
unix  2      [ ACC ]     STREAM     LISTENING     19788  2806/corosync
      @cpg
unix  2      [ ACC ]     STREAM     LISTENING     20022  2836/cib
      @cib_shm
unix  2      [ ACC ]     STREAM     LISTENING     19985  2840/pengine
      @pengine



Thanks in advance,

-- 
Best Regards,

Radoslaw Garbacz