[Pacemaker] fence_xvm / fence_virtd problem
Digimer
lists at alteeve.ca
Sun Jun 16 05:19:17 UTC 2013
I am confident (though never certain) that my issue is not the one
described in https://bugzilla.redhat.com/show_bug.cgi?id=880035
I can run omping from the host and guest for quite some time without issue:
host:
====
192.168.122.11 : unicast, seq=3207, size=69 bytes, dist=0, time=0.440ms
192.168.122.11 : multicast, seq=3207, size=69 bytes, dist=0, time=0.515ms
192.168.122.11 : waiting for response msg
192.168.122.11 : server told us to stop
192.168.122.11 : unicast, xmt/rcv/%loss = 3207/3207/0%,
min/avg/max/std-dev = 0.136/0.445/0.759/0.076
192.168.122.11 : multicast, xmt/rcv/%loss = 3207/3207/0%,
min/avg/max/std-dev = 0.150/0.505/0.813/0.084
====
guest:
====
192.168.122.1 : unicast, seq=3207, size=69 bytes, dist=0, time=0.445ms
192.168.122.1 : multicast, seq=3207, size=69 bytes, dist=0, time=0.478ms
^C
192.168.122.1 : unicast, xmt/rcv/%loss = 3207/3207/0%,
min/avg/max/std-dev = 0.117/0.388/0.637/0.065
192.168.122.1 : multicast, xmt/rcv/%loss = 3207/3206/0% (seq>=2 0%),
min/avg/max/std-dev = 0.124/0.418/0.882/0.070
====
digimer
On 06/16/2013 12:07 AM, Digimer wrote:
> Following that bug further; omping seems to work, if I am reading it right;
>
> Guest:
> ====
> [root at pcmk1 ~]# omping 192.168.122.1 192.168.122.11
> 192.168.122.1 : waiting for response msg
> 192.168.122.1 : waiting for response msg
> 192.168.122.1 : waiting for response msg
> 192.168.122.1 : waiting for response msg
> 192.168.122.1 : joined (S,G) = (*, 232.43.211.234), pinging
> 192.168.122.1 : unicast, seq=1, size=69 bytes, dist=0, time=0.308ms
> 192.168.122.1 : multicast, seq=1, size=69 bytes, dist=0, time=0.336ms
> 192.168.122.1 : unicast, seq=2, size=69 bytes, dist=0, time=0.373ms
> 192.168.122.1 : multicast, seq=2, size=69 bytes, dist=0, time=0.399ms
> 192.168.122.1 : unicast, seq=3, size=69 bytes, dist=0, time=0.480ms
> 192.168.122.1 : multicast, seq=3, size=69 bytes, dist=0, time=0.508ms
> 192.168.122.1 : unicast, seq=4, size=69 bytes, dist=0, time=0.384ms
> 192.168.122.1 : multicast, seq=4, size=69 bytes, dist=0, time=0.414ms
> 192.168.122.1 : unicast, seq=5, size=69 bytes, dist=0, time=0.431ms
> 192.168.122.1 : multicast, seq=5, size=69 bytes, dist=0, time=0.461ms
> ^C
> 192.168.122.1 : unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev =
> 0.308/0.395/0.480/0.065
> 192.168.122.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev =
> 0.336/0.424/0.508/0.065
> ====
>
> Host:
> ====
> lemass:/home/digimer# omping 192.168.122.1 192.168.122.11
> 192.168.122.11 : waiting for response msg
> 192.168.122.11 : joined (S,G) = (*, 232.43.211.234), pinging
> 192.168.122.11 : unicast, seq=1, size=69 bytes, dist=0, time=0.293ms
> 192.168.122.11 : unicast, seq=2, size=69 bytes, dist=0, time=0.398ms
> 192.168.122.11 : multicast, seq=2, size=69 bytes, dist=0, time=0.420ms
> 192.168.122.11 : unicast, seq=3, size=69 bytes, dist=0, time=0.470ms
> 192.168.122.11 : multicast, seq=3, size=69 bytes, dist=0, time=0.536ms
> 192.168.122.11 : unicast, seq=4, size=69 bytes, dist=0, time=0.526ms
> 192.168.122.11 : multicast, seq=4, size=69 bytes, dist=0, time=0.588ms
> 192.168.122.11 : unicast, seq=5, size=69 bytes, dist=0, time=0.443ms
> 192.168.122.11 : multicast, seq=5, size=69 bytes, dist=0, time=0.502ms
> 192.168.122.11 : waiting for response msg
> 192.168.122.11 : server told us to stop
>
> 192.168.122.11 : unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev
> = 0.293/0.426/0.526/0.088
> 192.168.122.11 : multicast, xmt/rcv/%loss = 5/4/19% (seq>=2 0%),
> min/avg/max/std-dev = 0.420/0.511/0.588/0.071
> ====
>
> digimer
>
> On 06/15/2013 11:56 PM, Digimer wrote:
>> Unfortunately, this didn't do it;
>>
>> Host;
>> ====
>> lemass:/home/digimer# fence_xvm -o list
>> pcmk1 83f6abdc-bb48-d794-4aca-13f091f32c8b on
>> pcmk2 2d778455-de7d-a9fa-994c-69d7b079fda8 on
>> ====
>>
>> Host: Other terminal
>> ====
>> lemass:/home/digimer# tcpdump -i virbr0 port zented
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>> decode
>> listening on virbr0, link-type EN10MB (Ethernet), capture size 65535
>> bytes
>> ====
>>
>> Guest;
>> ====
>> [root at pcmk1 ~]# fence_xvm -o list
>> Timed out waiting for response
>> Operation failed
>> ====
>>
>> Back on the host tcpdump terminal;
>> ====
>> lemass:/home/digimer# tcpdump -i virbr0 port zented
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>> decode
>> listening on virbr0, link-type EN10MB (Ethernet), capture size 65535
>> bytes
>> 23:55:06.803757 IP pcmk1.42078 > 225.0.0.12.zented: UDP, length 176
>> 23:55:08.806617 IP pcmk1.54274 > 225.0.0.12.zented: UDP, length 176
>> 23:55:10.809088 IP pcmk1.50136 > 225.0.0.12.zented: UDP, length 176
>> 23:55:12.811901 IP pcmk1.34244 > 225.0.0.12.zented: UDP, length 176
>> 23:55:14.814751 IP pcmk1.50129 > 225.0.0.12.zented: UDP, length 176
>> 23:55:16.817620 IP pcmk1.43212 > 225.0.0.12.zented: UDP, length 176
>> 23:55:18.820334 IP pcmk1.46487 > 225.0.0.12.zented: UDP, length 176
>> 23:55:20.823119 IP pcmk1.44280 > 225.0.0.12.zented: UDP, length 176
>> 23:55:22.825897 IP pcmk1.46148 > 225.0.0.12.zented: UDP, length 176
>> 23:55:24.828747 IP pcmk1.47381 > 225.0.0.12.zented: UDP, length 176
>> 23:55:26.831524 IP pcmk1.43129 > 225.0.0.12.zented: UDP, length 176
>> 23:55:28.834224 IP pcmk1.55455 > 225.0.0.12.zented: UDP, length 176
>> 23:55:30.836682 IP pcmk1.48112 > 225.0.0.12.zented: UDP, length 176
>> 23:55:32.839329 IP pcmk1.46622 > 225.0.0.12.zented: UDP, length 176
>> 23:55:34.842163 IP pcmk1.45112 > 225.0.0.12.zented: UDP, length 176
>> ====
>>
>> Host's firewall;
>>
>> ====
>> lemass:/home/digimer# iptables-save
>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 23:56:16 2013
>> *nat
>> :PREROUTING ACCEPT [615:151233]
>> :INPUT ACCEPT [72:15985]
>> :OUTPUT ACCEPT [818:69196]
>> :POSTROUTING ACCEPT [788:66896]
>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j
>> MASQUERADE --to-ports 1024-65535
>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j
>> MASQUERADE --to-ports 1024-65535
>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
>> COMMIT
>> # Completed on Sat Jun 15 23:56:16 2013
>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 23:56:16 2013
>> *mangle
>> :PREROUTING ACCEPT [88511:51215429]
>> :INPUT ACCEPT [41565:46046508]
>> :FORWARD ACCEPT [46167:4988181]
>> :OUTPUT ACCEPT [31831:3083584]
>> :POSTROUTING ACCEPT [78131:8099160]
>> -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM
>> --checksum-fill
>> COMMIT
>> # Completed on Sat Jun 15 23:56:16 2013
>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 23:56:16 2013
>> *filter
>> :INPUT ACCEPT [3160:898176]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [3663:430676]
>> -A INPUT -i virbr0 -j ACCEPT
>> -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
>> -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
>> -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
>> -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
>> -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate
>> RELATED,ESTABLISHED -j ACCEPT
>> -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
>> -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
>> -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
>> -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
>> COMMIT
>> # Completed on Sat Jun 15 23:56:16 2013
>> ====
>>
>> Fedora 19, fully updated to today.
>>
>> digimer
>>
>> On 06/15/2013 08:54 PM, Andrew Beekhof wrote:
>>> Apart form anything else, multicast continues to be broken in many
>>> kernels.
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=880035
>>>
>>> Run:
>>>
>>> tcpdump -i virbr0 port zented
>>>
>>> in another window and everything will magically start working.
>>>
>>>
>>> On 16/06/2013, at 3:26 AM, Digimer <lists at alteeve.ca> wrote:
>>>
>>>> Ah, I think it's a problem with the firewall rules on the host. Not
>>>> sure how to fix it though...
>>>>
>>>> lemass:/home/digimer# iptables-save
>>>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 13:26:33 2013
>>>> *nat
>>>> :PREROUTING ACCEPT [246583:89552160]
>>>> :INPUT ACCEPT [2335:362026]
>>>> :OUTPUT ACCEPT [11740:741351]
>>>> :POSTROUTING ACCEPT [11706:738225]
>>>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j
>>>> MASQUERADE --to-ports 1024-65535
>>>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j
>>>> MASQUERADE --to-ports 1024-65535
>>>> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
>>>> COMMIT
>>>> # Completed on Sat Jun 15 13:26:33 2013
>>>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 13:26:33 2013
>>>> *mangle
>>>> :PREROUTING ACCEPT [3250861:2486027770]
>>>> :INPUT ACCEPT [2557761:1301267981]
>>>> :FORWARD ACCEPT [444644:1094901100]
>>>> :OUTPUT ACCEPT [1919457:2636518995]
>>>> :POSTROUTING ACCEPT [2364615:3731498365]
>>>> -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM
>>>> --checksum-fill
>>>> COMMIT
>>>> # Completed on Sat Jun 15 13:26:33 2013
>>>> # Generated by iptables-save v1.4.16.2 on Sat Jun 15 13:26:33 2013
>>>> *filter
>>>> :INPUT ACCEPT [2557761:1301267981]
>>>> :FORWARD ACCEPT [0:0]
>>>> :OUTPUT ACCEPT [1919457:2636518995]
>>>> -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
>>>> -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
>>>> -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
>>>> -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
>>>> -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate
>>>> RELATED,ESTABLISHED -j ACCEPT
>>>> -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
>>>> -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
>>>> -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
>>>> -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
>>>> COMMIT
>>>> # Completed on Sat Jun 15 13:26:33 2013
>>>>
>>>> digimer
>>>>
>>>> On 06/15/2013 01:09 PM, Digimer wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm trying to play with pacemaker on fedora 19 (pre-release) and
>>>>> I am
>>>>> having trouble getting the guests to talk to the host.
>>>>>
>>>>> From the host, I can run;
>>>>>
>>>>> lemass:/home/digimer# fence_xvm -o list
>>>>> pcmk1 83f6abdc-bb48-d794-4aca-13f091f32c8b on
>>>>> pcmk2 2d778455-de7d-a9fa-994c-69d7b079fda8 on
>>>>>
>>>>> I can fence the guests from the host as well. However, I can not get
>>>>> the
>>>>> list (or fence) from the quests;
>>>>>
>>>>> [root at pcmk1 ~]# fence_xvm -o list
>>>>> Timed out waiting for response
>>>>> Operation failed
>>>>>
>>>>> I suspect a multicast issue, but so far as I can tell, multicast is
>>>>> enabled on the bridge;
>>>>>
>>>>> lemass:/home/digimer# ifconfig
>>>>> virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>>>>> inet 192.168.122.1 netmask 255.255.255.0 broadcast
>>>>> 192.168.122.255
>>>>> ether 52:54:00:da:90:a1 txqueuelen 0 (Ethernet)
>>>>> RX packets 103858 bytes 8514464 (8.1 MiB)
>>>>> RX errors 0 dropped 0 overruns 0 frame 0
>>>>> TX packets 151988 bytes 177742562 (169.5 MiB)
>>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>>>>
>>>>> vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>>>>> inet6 fe80::fc54:ff:feed:3701 prefixlen 64 scopeid
>>>>> 0x20<link>
>>>>> ether fe:54:00:ed:37:01 txqueuelen 500 (Ethernet)
>>>>> RX packets 212828 bytes 880551892 (839.7 MiB)
>>>>> RX errors 0 dropped 0 overruns 0 frame 0
>>>>> TX packets 225430 bytes 182955760 (174.4 MiB)
>>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>>>>
>>>>> vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>>>>> inet6 fe80::fc54:ff:fe45:e9ae prefixlen 64 scopeid
>>>>> 0x20<link>
>>>>> ether fe:54:00:45:e9:ae txqueuelen 500 (Ethernet)
>>>>> RX packets 4840 bytes 587902 (574.1 KiB)
>>>>> RX errors 0 dropped 0 overruns 0 frame 0
>>>>> TX packets 7495 bytes 899578 (878.4 KiB)
>>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>>>>
>>>>> I tried specifying the mcast address and port without success.
>>>>>
>>>>> The host's config is:
>>>>>
>>>>> lemass:/home/digimer# cat /etc/fence_virt.conf
>>>>> backends {
>>>>> libvirt {
>>>>> uri = "qemu:///system";
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> listeners {
>>>>> multicast {
>>>>> port = "1229";
>>>>> family = "ipv4";
>>>>> interface = "virbr0";
>>>>> address = "239.192.214.190";
>>>>> key_file = "/etc/cluster/fence_xvm.key";
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> fence_virtd {
>>>>> module_path = "/usr/lib64/fence-virt";
>>>>> backend = "libvirt";
>>>>> listener = "multicast";
>>>>> }
>>>>>
>>>>> The cluster forms and corosync is using multicast, so I am not sure if
>>>>> mcast really is the problem.
>>>>>
>>>>> Any tips/help?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>> --
>>>> Digimer
>>>> Papers and Projects: https://alteeve.ca/w/
>>>> What if the cure for cancer is trapped in the mind of a person
>>>> without access to education?
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>>
>
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
More information about the Pacemaker
mailing list