[Pacemaker] Trouble with DRBD mount

Andreas Kurz andreas at hastexo.com
Thu Feb 28 18:10:05 EST 2013


On 2013-02-28 13:19, senrabdet at aol.com wrote:
> Hi All:
> 
> We are stuck trying to get pacemaker to work with DRBD, and having tried
> various alternatives can't get our "drbd1" to mount and get some errors.
> 
> NOTE:  we are trying to get pacemaker to work with an existing Encrypted
> RAID1 LVM setup - is this impossible or a "just plain bad idea"?   We
> were thinking we'd like the potential advantages of local RAID on each
> box as well as the Internet RAID & failover provided by DRBD/pacemaker.
>  We're using Debian Squeeze.  Per various instructions, we've disabled
> the DRBD boot init (update-rc.d -f drbd remove) and set the LVM filter
> to filter = [ "a|drbd.*|", "r|.*|" ].

so you only allow scanning for LVM signatures on DRBD ... that needs to
be in Primary mode before ....

> 
> FYI - we've commented out the LVM mount "/dev/vg2/vserverLV" in our
> fstab, and consistently seem to need to do this to avoid a boot error.
> 
> We think DRBD works until we add in the pacemaker steps (i.e.,
> "dev/drbd1" mounts at boot; we can move related data from server1 to
> server2 back and forth, though need to use the command line to
> accomplish this).  We've seen various statements on the net that suggest
> it is viable to use a "mapper" disk choice in drbd.conf.  Also, if we
> start by configuring Pacemaker for a simple IP failover, that works
> (i.e., no errors, we can ping via the fail over address) but stops
> working when we add in the DRBD primatives and related statements.  Our
> suspicion (other than maybe "you can't do this with existing RAID") is
> that we're using the wrong "disk" statement in our drbd.conf or maybe in
> our "primitive fs_vservers" statement, though we've tried lots of
> alternatives and this is the same drbd.conf we use before adding in
> Pacemaker and it seems to work at that point.
> 
> Lastly, while various config statements refer to "vservers", we have not
> gotten to the point of trying to add any data to the DRBD devices other
> than a few text files that have disappeared since doing our "crm" work.
> 
> Any help appreciated!  Thanks, Ted
> 
> CONFIGS/LOGS
> 
> A) drbd.conf
> 
> global { usage-count no; }
> common { syncer { rate 100M; } }
> #original
> resource r1 {
>         protocol C;
> startup {
>                 wfc-timeout  15;
>                 degr-wfc-timeout 60;
>         }
>         device /dev/drbd1 minor 1;
>           disk /dev/vg2/vserverLV;

so vg2/vserverLV is the lower-level device for DRBD, simply let vg2 be
automatically activated and forget that LVM filter thing you did, that
is only needed for vgs sitting _on_ DRBD, not below.

>         meta-disk internal;
> 
> # following 2 definition are equivalent
>         on server1 {
>                 address 192.168.1.129:7801;
>                  disk /dev/vg2/vserverLV;
>         }
>         on server2 {
>                 address 192.168.1.128:7801;
>                  disk /dev/vg2/vserverLV;
> #disk /dev/mapper/md2_crypt;
>         }
> 
> #       floating 192.168.5.41:7801;
> #       floating 192.168.5.42:7801;
>          net {
> cram-hmac-alg sha1;
>                 shared-secret "secret";
>                   after-sb-0pri discard-younger-primary;
> #discard-zero-changes;
>                   after-sb-1pri discard-secondary;
>                   after-sb-2pri call-pri-lost-after-sb;
>         }
> }
> 
> 
> B) Pacemaker Config
> 
> crm configure show
> node server1
> node server2
> primitive app_ip ocf:heartbeat:IPaddr \
> params ip="192.168.1.152" \
> op monitor interval="30s"
> primitive drbd ocf:linbit:drbd \
> params drbd_resource="r1" \
> op start interval="0" timeout="240" \
> op stop interval="0" timeout="100" \
> op monitor interval="59s" role="Master" timeout="30s" \
> op monitor interval="60s" role="Slave" timeout="30s"
> primitive fs_vservers ocf:heartbeat:Filesystem \
> params device="/dev/drbd1" directory="/vservers" fstype="ext4" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="120"
> primitive vg2 ocf:heartbeat:LVM \
> params volgrpname="vg2" exclusive="true" \

simply remove all that LVM things from your pacemaker configuration

> op start interval="0" timeout="30" \
> op stop interval="0" timeout="30"
> group lvm app_ip vg2 fs_vservers

ouch .. a group called "lvm", am I the only one who thinks this is
confusing?

> ms ms_drbd drbd \
> meta master-node-max="1" clone-max="2" clone-node-max="1"
> globally-unique="false" notify="true" target-role="Master"
> location drbd_on_node1 ms_drbd \
> rule $id="drbd_on_node1-rule" $role="master" 100: #uname eq server1
> colocation vserver-deps inf: ms_drbd:Master lvm

wrong direction  .. you want the group follow the DRBD master

Regards,
Andreas

-- 
Need help with Pacemaker?
http://www.hastexo.com/now


> order app_on_drbd inf: ms_drbd:promote lvm:start
> property $id="cib-bootstrap-options" \
> dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no-quorum-policy="ignore"
> 
> 
> 
> C)  crm status results (with errors)
> Last updated: Wed Feb 27 19:05:57 2013
> Stack: openais
> Current DC: server1 - partition with quorum
> Version: 1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b
> 2 Nodes configured, 2 expected votes
> 2 Resources configured.
> ============
> 
> Online: [ server2 server1 ]
> 
> 
> Migration summary:
> * Node server2: 
>    drbd:1: migration-threshold=1000000 fail-count=1000000
> * Node server1: 
>    drbd:0: migration-threshold=1000000 fail-count=1000000
> 
> Failed actions:
>     drbd:1_start_0 (node=server2, call=8, rc=-2, status=Timed Out):
> unknown exec error
>     drbd:0_start_0 (node=server1, call=6, rc=-2, status=Timed Out):
> unknown exec error
> 
> D)  Mount
> 
> /dev/mapper/vg1-root on / type ext4 (rw,errors=remount-ro)
> tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> udev on /dev type tmpfs (rw,mode=0755)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
> /dev/md0 on /boot type ext4 (rw)
> /dev/mapper/vg1-home on /home type ext4 (rw)
> /dev/mapper/vg1-tmp on /tmp type ext4 (rw)
> /dev/mapper/vg1-usr on /usr type ext4 (rw)
> /dev/mapper/vg1-var on /var type ext4 (rw)
> fusectl on /sys/fs/fuse/connections type fusectl (rw)
> 
> 
> E)  fstab
> 
> # /etc/fstab: static file system information.
> #
> # Use 'blkid' to print the universally unique identifier for a
> # device; this may be used with UUID= as a more robust way to name devices
> # that works even if disks are added and removed. See fstab(5).
> #
> # <file system> <mount point>   <type>  <options>       <dump>  <pass>
> proc            /proc           proc    defaults        0       0
> /dev/mapper/vg1-root /               ext4    errors=remount-ro 0       1
> # /boot was on /dev/md0 during installation
> UUID=25829c6c-164c-4a1e-9e84-6bab180e38f4 /boot           ext4  
>  defaults        0       2
> /dev/mapper/vg1-home /home           ext4    defaults        0       2
> /dev/mapper/vg1-tmp /tmp            ext4    defaults        0       2
> /dev/mapper/vg1-usr /usr            ext4    defaults        0       2
> /dev/mapper/vg1-var /var            ext4    defaults        0       2
> #/dev/mapper/vg2-vserverLV /vservers       ext4    defaults        0       2
> /dev/mapper/vg1-swap none            swap    sw              0       0
> /dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
> /dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0
> /dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0
> 
> 
> F)  fdisk -l
> 
> Disk /dev/sda: 160.0 GB, 160041885696 bytes
> 255 heads, 63 sectors/track, 19457 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x0007c7a2
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1   *           1          61      487424   fd  Linux raid
> autodetect
> Partition 1 does not end on cylinder boundary.
> /dev/sda2              61        1885    14648320   fd  Linux raid
> autodetect
> /dev/sda3            1885        3101     9765888   fd  Linux raid
> autodetect
> 
> Disk /dev/sdb: 203.9 GB, 203928109056 bytes
> 255 heads, 63 sectors/track, 24792 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x0008843c
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdb1   *           1          61      487424   fd  Linux raid
> autodetect
> Partition 1 does not end on cylinder boundary.
> /dev/sdb2              61        1885    14648320   fd  Linux raid
> autodetect
> /dev/sdb3            1885        3101     9765888   fd  Linux raid
> autodetect
> 
> Disk /dev/md0: 499 MB, 499109888 bytes
> 2 heads, 4 sectors/track, 121853 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/md1: 15.0 GB, 14998757376 bytes
> 2 heads, 4 sectors/track, 3661806 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x08040000
> 
> 
> Disk /dev/md2: 9999 MB, 9999147008 bytes
> 2 heads, 4 sectors/track, 2441198 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x08040000
> 
> 
> Disk /dev/dm-0: 15.0 GB, 14997704704 bytes
> 255 heads, 63 sectors/track, 1823 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-1: 3997 MB, 3997171712 bytes
> 255 heads, 63 sectors/track, 485 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-2: 1996 MB, 1996488704 bytes
> 255 heads, 63 sectors/track, 242 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-3: 1996 MB, 1996488704 bytes
> 255 heads, 63 sectors/track, 242 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-4: 3997 MB, 3997171712 bytes
> 255 heads, 63 sectors/track, 485 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-5: 1996 MB, 1996488704 bytes
> 255 heads, 63 sectors/track, 242 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-6: 499 MB, 499122176 bytes
> 255 heads, 63 sectors/track, 60 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> 
> Disk /dev/dm-7: 9998 MB, 9998094336 bytes
> 255 heads, 63 sectors/track, 1215 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000000
> 
> G)  syslog excerpt
> 
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) Command '
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) drbdsetup
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) 1
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) disk
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) /dev/vg2/vserverLV
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) /dev/vg2/vserverLV
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) internal
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) --set-defaults
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr)  
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) --create-device
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) ' terminated with exit code 20
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) drbdadm attach r1: exited with code 20
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Called drbdadm -c
> /etc/drbd.conf --peer server2 up r1
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Exit code 1
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Command output: 
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stdout) 
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) Can not open device '/dev/vg2/vserverLV': No
> such file or directory
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) Command 'drbdsetup 1 disk /dev/vg2/vserverLV 
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) /dev/vg2/vserverLV internal --set-defaults
> --create-device' terminated with exit code 20#012drbdadm attach r1:
> exited with code 20
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Called drbdadm -c
> /etc/drbd.conf --peer server2 up r1
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Exit code 1
> Feb 27 06:36:22 server1 drbd[2329]: ERROR: r1: Command output: 
> Feb 27 06:36:22 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stdout) 
> Feb 27 06:36:23 server1 lrmd: [1705]: info: RA output:
> (p_drbd_r1:0:start:stderr) Can not open device '/dev/vg2/vserverLV': No
> such file or directory
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 






More information about the Pacemaker mailing list