[Pacemaker] Problem: No space left on device.

Kees chkoehoorn at live.nl
Tue Mar 9 13:39:12 UTC 2010 

Hello Andrew/Thomas,

Stupid me.... it was indeed an inode problem, nothing to do with the 
cluster-software. The /var/lib/pengine directory was a bit full.
//I configured the pe-*-serires-max options so this should not happen 
again, thanks.


> On Fri, Mar 5, 2010 at 3:38 PM, Kees<chkoehoorn at live.nl>  wrote:
>    
>> Hi,
>>
>> When i start the cluster software with /etc/init.d/corosync start, i see the
>> whole stack in my processlist:
>>
>> 31838 ?        Ssl    0:06 /usr/sbin/corosync
>> 31849 ?        SLs    0:00  \_ /usr/lib/heartbeat/stonithd
>> 31850 ?        S      0:02  \_ /usr/lib/heartbeat/cib
>> 31851 ?        S      0:01  \_ /usr/lib/heartbeat/lrmd
>> 31852 ?        S      0:00  \_ /usr/lib/heartbeat/attrd
>> 31853 ?        S      0:00  \_ /usr/lib/heartbeat/pengine
>> 31854 ?        S      0:00  \_ /usr/lib/heartbeat/crmd
>>
>> I looks like everything is running, but there is a problem:
>>
>> daemon.log:Mar  5 11:54:24 test1 cib: [23150]: ERROR: write_xml_file: Cannot
>> open /var/lib/heartbeat/crm/cib.qFnnLt for writing: No space left on device
>> (28)
>> daemon.log:Mar  5 11:55:27 test1 pengine: [23145]: ERROR: write_xml_file:
>> Cannot open /var/lib/pengine/pe-warn-418392.bz2 for writing: No space left
>> on device (28)
>>      
> You might want to set the pe-*-series-max options to limit the amount
> of space used to store old PE inputs (used for debugging)
> Looks like you have quite a few.
>
>      </parameter>
>      <parameter name="pe-error-series-max" unique="0">
>        <shortdesc lang="en">The number of PE inputs resulting in ERRORs
> to save</shortdesc>
>        <content type="integer" default="-1"/>
>        <longdesc lang="en">Zero to disable, -1 to store unlimited.</longdesc>
>      </parameter>
>      <parameter name="pe-warn-series-max" unique="0">
>        <shortdesc lang="en">The number of PE inputs resulting in
> WARNINGs to save</shortdesc>
>        <content type="integer" default="-1"/>
>        <longdesc lang="en">Zero to disable, -1 to store unlimited.</longdesc>
>      </parameter>
>      <parameter name="pe-input-series-max" unique="0">
>        <shortdesc lang="en">The number of other PE inputs to save</shortdesc>
>        <content type="integer" default="-1"/>
>        <longdesc lang="en">Zero to disable, -1 to store unlimited.</longdesc>
>      </parameter>
>
>
>    
>> daemon.log:Mar  5 11:55:28 test1 pengine: [23145]: ERROR: write_xml_file:
>> Cannot open /var/lib/pengine/pe-warn-418393.bz2 for writing: No space left
>> on device (28)
>> daemon.log:Mar  5 11:55:28 test1 pengine: [23145]: ERROR: write_xml_file:
>> Cannot open /var/lib/pengine/pe-warn-418394.bz2 for writing: No space left
>> on device (28)
>> daemon.log:Mar  5 11:55:28 test1 lrmd: [23143]: info: RA output:
>> (ip_storage:start:stderr) info: Could not open pid-file
>> [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space left on
>> device
>> daemon.log:Mar  5 11:55:28 test1 send_arp: [23358]: info: Could not open
>> pid-file [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space
>> left on device
>> daemon.log:Mar  5 12:13:18 test1 cib: [24900]: ERROR: write_xml_file: Cannot
>> open /var/lib/heartbeat/crm/cib.2rfyDF for writing: No space left on device
>> (28)
>> daemon.log:Mar  5 12:19:11 test1 lrmd: [24894]: info: RA output:
>> (ip_storage:start:stderr) info: Could not open pid-file
>> [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space left on
>> device
>> daemon.log:Mar  5 12:19:11 test1 send_arp: [26746]: info: Could not open
>> pid-file [/var/run/heartbeat/rsctmp/send_arp/send_arp-172.16.0.4]: No space
>> left on device
>> daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:start:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:start:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:start:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:47 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:promote:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:promote:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:notify:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>> daemon.log:Mar  5 12:25:48 test1 lrmd: [24894]: info: RA output:
>> (drbd_websites:0:monitor:stderr) symlink(/etc/drbd.conf,
>> /var/lib/drbd//drbd-minor-0.conf): No space left on device
>>
>> Somehow my /var partion is not writeable anymore. When i try it myself with
>> a 'touch testfile' i get the same error:
>>
>> touch: cannot touch `testfile': No space left on device
>>
>> When i stop the cluster, i can write again to /var. I can't find the
>> problem, what is going wrong here?
>>
>> Debian leny
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sda5             942M  116M  779M  13% /
>> /dev/sda1             942M   38M  857M   5% /boot
>> /dev/sda6             942M   18M  877M   2% /home
>> /dev/sda10            1.9G   35M  1.8G   2% /tmp
>> /dev/sda7             1.9G  593M  1.2G  34% /usr
>> /dev/sda8             1.9G  894M  888M  51% /var
>> /dev/sda9             1.9G   57M  1.7G   4% /var/log
>> /dev/drbd0            102G  188M   97G   1% /websites
>>
>> corosync_1.2.0-1_i386.deb
>> pacemaker_1.0.7+hg20100203-1_i386.deb
>>
>> I use the Debian-packages from madkiss.
>>
>>
>> Greeting,
>>
>> Kees
>>
>>
>>
>>
>> Thanks in advance for your help.
>>
>> Kees Koehoorn
>>
>> _______________________________________________
>> Pacemaker mailing list
>> Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>>      
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
>
>    

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20100309/2a427be9/attachment-0002.htm>

More information about the Pacemaker mailing list