Diff hostcache... no /tmp/highload/*/hostcache :/
Diff members.txt... OK
Diff cib.xml... OK
Diff crm_mon.txt... OK
Diff ha.cf... no /tmp/highload/*/ha.cf :/
Diff logd.cf... no /tmp/highload/*/logd.cf :/
Diff sysinfo.txt... OK
WARN: problem with permissions/ownership at INACD-DB-SVR1-TEST:
/var/lib/heartbeat/ccm wrong type or doesn't exist
/var/lib/heartbeat/pengine wrong type or doesn't exist
WARN: problem with permissions/ownership at INACD-DB-SVR2-TEST:
/var/lib/heartbeat/ccm wrong type or doesn't exist
/var/lib/heartbeat/pengine wrong type or doesn't exist

Log patterns:
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[14] on ocf::mysql::mysql-master for client 3685, its parameters: CRM_meta_interval=[10000] socket=[/tmp/mysql-acd-test.sock] test_passwd=[testcluster] binary=[/packages/11.1/mysql/libexec/mysqld] group=[mysql] additional_parameters=[--skip-external-locking] test_user=[clustertestuser] CRM_meta_timeout=[10000] crm_feature_set=[3.0.1] pid=[/packages/11.1/mysql/var/INACD-DB-SVR1-TEST.pid] test_table=[ACD_Configuration.ACD_Mandator] user=[mysql] CRM_meta_name=[monitor] config=[/packages/11.1/
Feb  4 15:35:58 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:19 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:37:50 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:38:10 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:57 INACD-DB-SVR1-TEST [30503]: ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: info: RA output: (mysql-master:monitor:stderr) 2010/02/04_15:40:27 ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation stop[20] on ocf::pingd::pingd:0 for client 3685, its parameters: dampen=[10] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_clone=[0] 
Feb  4 15:40:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: process_lrm_event: LRM operation pingd:0_stop_0 (call=20, status=4, cib-update=72, confirmed=true) Error
Feb  4 15:41:06 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: TrackedProcTimeoutFunction: bad pinfo in call (pid 30503)
Feb  4 15:43:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: print_elem: Aborting transition, action lost: [Action 3]: In-flight (id: mysql-master_stop_0, loc: INACD-DB-SVR1-TEST, priority: 0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: native_add_running: Resource stonith::apcmastersnmp:apcstonith-INACD-DB-SVR2-TEST appears to be active on 2 nodes.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: process_pe_message: Transition 0: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-2.bz2
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[14] on ocf::mysql::mysql-master for client 3685, its parameters: CRM_meta_interval=[10000] socket=[/tmp/mysql-acd-test.sock] test_passwd=[testcluster] binary=[/packages/11.1/mysql/libexec/mysqld] group=[mysql] additional_parameters=[--skip-external-locking] test_user=[clustertestuser] CRM_meta_timeout=[10000] crm_feature_set=[3.0.1] pid=[/packages/11.1/mysql/var/INACD-DB-SVR1-TEST.pid] test_table=[ACD_Configuration.ACD_Mandator] user=[mysql] CRM_meta_name=[monitor] config=[/packages/11.1/
Feb  4 15:35:58 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:19 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:37:50 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:38:10 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:57 INACD-DB-SVR1-TEST [30503]: ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: info: RA output: (mysql-master:monitor:stderr) 2010/02/04_15:40:27 ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation stop[20] on ocf::pingd::pingd:0 for client 3685, its parameters: dampen=[10] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_clone=[0] 
Feb  4 15:40:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: process_lrm_event: LRM operation pingd:0_stop_0 (call=20, status=4, cib-update=72, confirmed=true) Error
Feb  4 15:41:06 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: TrackedProcTimeoutFunction: bad pinfo in call (pid 30503)
Feb  4 15:43:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: print_elem: Aborting transition, action lost: [Action 3]: In-flight (id: mysql-master_stop_0, loc: INACD-DB-SVR1-TEST, priority: 0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: native_add_running: Resource stonith::apcmastersnmp:apcstonith-INACD-DB-SVR2-TEST appears to be active on 2 nodes.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: process_pe_message: Transition 0: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-2.bz2
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[14] on ocf::mysql::mysql-master for client 3685, its parameters: CRM_meta_interval=[10000] socket=[/tmp/mysql-acd-test.sock] test_passwd=[testcluster] binary=[/packages/11.1/mysql/libexec/mysqld] group=[mysql] additional_parameters=[--skip-external-locking] test_user=[clustertestuser] CRM_meta_timeout=[10000] crm_feature_set=[3.0.1] pid=[/packages/11.1/mysql/var/INACD-DB-SVR1-TEST.pid] test_table=[ACD_Configuration.ACD_Mandator] user=[mysql] CRM_meta_name=[monitor] config=[/packages/11.1/
Feb  4 15:35:58 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:19 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:37:50 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:38:10 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:57 INACD-DB-SVR1-TEST [30503]: ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: info: RA output: (mysql-master:monitor:stderr) 2010/02/04_15:40:27 ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation stop[20] on ocf::pingd::pingd:0 for client 3685, its parameters: dampen=[10] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_clone=[0] 
Feb  4 15:40:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: process_lrm_event: LRM operation pingd:0_stop_0 (call=20, status=4, cib-update=72, confirmed=true) Error
Feb  4 15:41:06 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: TrackedProcTimeoutFunction: bad pinfo in call (pid 30503)
Feb  4 15:43:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: print_elem: Aborting transition, action lost: [Action 3]: In-flight (id: mysql-master_stop_0, loc: INACD-DB-SVR1-TEST, priority: 0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: native_add_running: Resource stonith::apcmastersnmp:apcstonith-INACD-DB-SVR2-TEST appears to be active on 2 nodes.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: process_pe_message: Transition 0: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-2.bz2
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:34:59 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[14] on ocf::mysql::mysql-master for client 3685, its parameters: CRM_meta_interval=[10000] socket=[/tmp/mysql-acd-test.sock] test_passwd=[testcluster] binary=[/packages/11.1/mysql/libexec/mysqld] group=[mysql] additional_parameters=[--skip-external-locking] test_user=[clustertestuser] CRM_meta_timeout=[10000] crm_feature_set=[3.0.1] pid=[/packages/11.1/mysql/var/INACD-DB-SVR1-TEST.pid] test_table=[ACD_Configuration.ACD_Mandator] user=[mysql] CRM_meta_name=[monitor] config=[/packages/11.1/
Feb  4 15:35:58 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:19 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:36:32 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:37:20 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[12] on ocf::pingd::pingd:0 for client 3685, its parameters: CRM_meta_interval=[30000] dampen=[10] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] CRM_meta_clone=[0] 
Feb  4 15:37:50 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:38:10 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:16 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:39:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation monitor[17] on ocf::IPaddr2::mysql-master-ip for client 3685, its parameters: CRM_meta_interval=[10000] ip=[10.2.50.24] cidr_netmask=[24] CRM_meta_timeout=[10000] broadcast=[10.2.50.255] crm_feature_set=[3.0.1] CRM_meta_name=[monitor] nic=[eth0] 
Feb  4 15:39:57 INACD-DB-SVR1-TEST [30503]: ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: info: RA output: (mysql-master:monitor:stderr) 2010/02/04_15:40:27 ERROR: MySQL  monitor failed:
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: perform_ra_op::3041: fork: Cannot allocate memory
Feb  4 15:40:56 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: unable to perform_ra_op on operation stop[20] on ocf::pingd::pingd:0 for client 3685, its parameters: dampen=[10] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] host_list=[10.2.50.11 10.2.50.40 10.2.50.8] interval=[5] CRM_meta_globally_unique=[false] CRM_meta_clone=[0] 
Feb  4 15:40:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: process_lrm_event: LRM operation pingd:0_stop_0 (call=20, status=4, cib-update=72, confirmed=true) Error
Feb  4 15:41:06 INACD-DB-SVR1-TEST lrmd: [3682]: ERROR: TrackedProcTimeoutFunction: bad pinfo in call (pid 30503)
Feb  4 15:43:56 INACD-DB-SVR1-TEST crmd: [3685]: ERROR: print_elem: Aborting transition, action lost: [Action 3]: In-flight (id: mysql-master_stop_0, loc: INACD-DB-SVR1-TEST, priority: 0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: native_add_running: Resource stonith::apcmastersnmp:apcstonith-INACD-DB-SVR2-TEST appears to be active on 2 nodes.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: process_pe_message: Transition 0: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-2.bz2
