Diff hostcache... --- /tmp/report/ha-test1/hostcache	2010-03-22 10:10:58.000000000 +0100
+++ /tmp/report/ha-test2/hostcache	2010-03-22 15:38:41.000000000 +0100
@@ -1,2 +1,2 @@
-ha-test1	00630d01-9d39-49df-a804-554c504c5477	100	
 ha-test2	f6f5fee1-46d3-4e1f-941a-bc66f0adbb0f	100	
+ha-test1	00630d01-9d39-49df-a804-554c504c5477	100	

Diff members.txt... OK
Diff cib.xml... OK
Diff crm_mon.txt... OK
Diff corosync.conf... --- /tmp/report/ha-test1/corosync.conf	2010-03-19 17:30:00.000000000 +0100
+++ /tmp/report/ha-test2/corosync.conf	2010-03-22 19:12:17.000000000 +0100
@@ -39,7 +39,7 @@
  	interface {
 		# The following values need to be set based on your environment 
 		ringnumber: 0
-		bindnetaddr: 10.133.99.91
+		bindnetaddr: 10.133.99.92
 		mcastaddr: 226.94.1.1
 		mcastport: 5405
 	}

Diff logd.cf... no /tmp/report/*/logd.cf :/
Diff sysinfo.txt... --- /tmp/report/ha-test1/sysinfo.txt	2010-03-24 11:59:08.000000000 +0100
+++ /tmp/report/ha-test2/sysinfo.txt	2010-03-24 11:59:06.000000000 +0100
@@ -3,7 +3,7 @@
 cluster-glue: 1.0.2 (9bcd134f1ebff7baf80f4b21c3b5f620b0ee976e)
 resource-agents: # Build version: 61203c6ef710e35660f0d76047561877963d3a2f
 CRM Version: 1.0.7 (54d7869bfe3691eb723b1d47810e5585d8246b58)
- 1.0.7+hg20100203-1  1.1.1-1  1.1.1-1 1.2.0-1 1.2.0-1 1.0.3-1~bpo50+1  1:3.0.2-1~bpo50+1  1:3.0.2-1~bpo50+1 1.4.2-1.1~bpo50+1    2.02.53-2.1~bpo50+1 2.20081102-1+lenny1 3.0.4-2.1~bpo50+1Platform: Linux
+ 1.0.7+hg20100203-1 1.1.1-1  1.1.1-1 1.2.0-1 1.2.0-1 1.0.3-1~bpo50+1  1:3.0.2-1~bpo50+1  1:3.0.2-1~bpo50+1 1.4.2-1.1~bpo50+1    2.02.53-2.1~bpo50+1 2.20081102-1+lenny1 3.0.4-2.1~bpo50+1Platform: Linux
 Kernel release: 2.6.33.1
 Architecture: i686
 Distribution: /etc/debian_version 5.0.4

WARN: problem with permissions/ownership at ha-test1:
wrong permissions or ownership for /var/lib/pengine:
drwxr-xr-x 2 hacluster haclient 20480 Mar 24 11:57 /var/lib/pengine
WARN: problem with permissions/ownership at ha-test2:
wrong permissions or ownership for /var/lib/pengine:
drwxr-xr-x 2 hacluster haclient 12288 Mar 24 10:45 /var/lib/pengine

Log patterns:
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Command output: 
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="844" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Command output: 
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="852" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="848" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="856" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Command output: 
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="844" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Command output: 
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="852" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="848" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="856" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
