[Pacemaker] Problem: No space left on device.

[Kees]

Hi,

When i start the cluster software with /etc/init.d/corosync start, i see 
the whole stack in my processlist:

31838 ?        Ssl    0:06 /usr/sbin/corosync
31849 ?        SLs    0:00  \_ /usr/lib/heartbeat/stonithd
31850 ?        S      0:02  \_ /usr/lib/heartbeat/cib
31851 ?        S      0:01  \_ /usr/lib/heartbeat/lrmd
31852 ?        S      0:00  \_ /usr/lib/heartbeat/attrd
31853 ?        S      0:00  \_ /usr/lib/heartbeat/pengine
31854 ?        S      0:00  \_ /usr/lib/heartbeat/crmd

I looks like everything is running, but there is a problem:

daemon.log:Mar  5 11:54:24 test1 cib: [23150]: ERROR: write_xml_file: 
Cannot open /var/lib/heartbeat/crm/cib.qFnnLt for writing: No space left 
on device (28)
daemon.log:Mar  5 11:55:27 test1 pengine: [23145]: ERROR: 
write_xml_file: Cannot open /var/lib/pengine/pe-warn-418392.bz2 for 
writing: No space left on device (28)
daemon.log:Mar  5 11:55:28 test1 pengine: [23145]: ERROR: 
write_xml_file: Cannot open /var/lib/pengine/pe-warn-418393.bz2 for 
writing: No space left on device (28)
daemon.log:Mar  5 11:55:28 test1 pengine: [23145]: ERROR: 
write_xml_file: Cannot open /var/lib/pengine/pe-warn-418394.bz2 for 
writing: No space left on device (28)
daemon.log:Mar  5 11:55:28 test1 lrmd: [23143]: info: RA output: 
(ip_storage:start:stderr) info: Could not open pid-file 
[/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space left 
on device
daemon.log:Mar  5 11:55:28 test1 send_arp: [23358]: info: Could not open 
pid-file [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No 
space left on device
daemon.log:Mar  5 12:13:18 test1 cib: [24900]: ERROR: write_xml_file: 
Cannot open /var/lib/heartbeat/crm/cib.2rfyDF for writing: No space left 
on device (28)
daemon.log:Mar  5 12:19:11 test1 lrmd: [24894]: info: RA output: 
(ip_storage:start:stderr) info: Could not open pid-file 
[/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space left 
on device
daemon.log:Mar  5 12:19:11 test1 send_arp: [26746]: info: Could not open 
pid-file [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No 
space left on device
daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:start:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:start:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:start:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:promote:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:promote:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device
daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output: 
(drbd_websites:0:monitor:stderr) symlink(/etc/drbd.conf, 
/var/lib/drbd//drbd-minor-0.conf): No space left on device

Somehow my /var partion is not writeable anymore. When i try it myself 
with a 'touch testfile' i get the same error:

touch: cannot touch `testfile': No space left on device

When i stop the cluster, i can write again to /var. I can't find the 
problem, what is going wrong here?

Debian leny

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             942M  116M  779M  13% /
/dev/sda1             942M   38M  857M   5% /boot
/dev/sda6             942M   18M  877M   2% /home
/dev/sda10            1.9G   35M  1.8G   2% /tmp
/dev/sda7             1.9G  593M  1.2G  34% /usr
/dev/sda8             1.9G  894M  888M  51% /var
/dev/sda9             1.9G   57M  1.7G   4% /var/log
/dev/drbd0            102G  188M   97G   1% /websites

corosync_1.2.0-1_i386.deb
pacemaker_1.0.7+hg20100203-1_i386.deb

I use the Debian-packages from madkiss.


Greeting,

Kees




Thanks in advance for your help.

Kees Koehoorn