settingsLogin | Registersettings

[openstack-dev] [octavia] haproxy fails to receive datagram

0 votes

Hi, all,

I encounter some problems when using Octavia. After installing octavia with
devstack, I create a load balancer named lb1 (VIP: 10.0.1.9, IP of VRRP
port: 10.0.1.3) for a subnet (10.0.1.0/24), then a listener, a pool, and
two members. All the resources are created successfully. The two members
(VMs) reside in the same subnet, whose IP are 10.0.1.6 and 10.0.1.7,
respectively. To simulate a web server in each VM, I run "while true; do
echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $VMIP" | sudo nc -l -p 80;done"
to listen on port 80 and return the VM's IP if the request is accepted. I
run "sudo ip netns exec qdhcp-XXXX curl -v $VM
IP" to send requests to VMs,
the "web servers" in VMs work (already added corresponding security rules
to the VMs). Then I tried to run "sudo ip netns exec qdhcp-XXXX curl -v
$VIP" to send requests, the VMs do not respond, and finally returns a
timeout error.

The configuration details in local.conf are as follows.

[[local|localrc]]

DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
ADMIN_PASSWORD=password

HOST_IP=192.168.56.9

LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=/opt/stack/logs

# Neutron LBaaS
enable_plugin neutron-lbaas https://github.com/openstack/neutron-lbaas.git
https://github.com/openstack/neutron-lbaas.git

enable_plugin octavia https://github.com/openstack/octavia.git
https://github.com/openstack/octavia.git

ENABLED_SERVICES+=,q-lbaasv2
ENABLED_SERVICES+=,octavia,o-cw,o-hk,o-hm,o-api

disable_service horizon
disable_service tempest

To investigate the source of the error, I logon to the amphora. The details
of interfaces of amphora_haproxy network namespace are as follows.

1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
* link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00*
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast state
UP group default qlen 1000

* link/ether fa:16:3e:42:bf:d9 brd ff:ff:ff:ff:ff:ff*
* inet 10.0.1.3/24 http://10.0.1.3/24 brd 10.0.1.255 scope global eth1*
* validlft forever preferredlft forever*
* inet 10.0.1.9/24 http://10.0.1.9/24 brd 10.0.1.255 scope global
secondary eth1:0*
* validlft forever preferredlft forever*
* inet6 fe80::f816:3eff:fe42:bfd9/64 scope link*
* validlft forever preferredlft forever*

So I run "sudo ip netns exec amphora-haproxy tcpdump -i eth1 -nn 'tcp'" to
check whether amphora receive the request. The details are as follows.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C06:11:49.048973 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28032309 ecr
0,nop,wscale 7], length 0

06:11:50.031976 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28032559 ecr
0,nop,wscale 7], length 0

06:11:52.026565 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28033060 ecr
0,nop,wscale 7], length 0

06:11:56.002577 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28034062 ecr
0,nop,wscale 7], length 0

06:12:03.909721 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28036064 ecr
0,nop,wscale 7], length 0

Based on the trace, we can see that amphora do receive the request, but
haproxy does not send handshake datagram to respond. Then, to see whether
haproxy in the amphora listens on the right IP and port, I print
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg in the
console and see the following info.

# Configuration for lb1
global
* daemon*
* user nobody*
* log /dev/log local0*
* log /dev/log local1 notice*
* stats socket
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57.sock mode 0666 level
user*

defaults
* log global*
* retries 3*
* option redispatch*
* timeout connect 5000*
* timeout client 50000*
* timeout server 50000*

frontend ec5ee7c5-7474-424f-9f44-71b338cf3e57
* option httplog*
* bind 10.0.1.9:80 http://10.0.1.9:80*
* mode http*
* default_backend 40537c80-979d-49c9-b3ae-8504812c0f42*

backend 40537c80-979d-49c9-b3ae-8504812c0f42
* mode http*
* balance roundrobin*
* server 73dc9a1d-1e92-479b-a6f3-8debd0ea17b8 10.0.1.6:80
http://10.0.1.6:80 weight 1*
* server 4cdca33f-9cde-4ac2-a5bd-550d3e65f0f2 10.0.1.7:80
http://10.0.1.7:80 weight 1*

Next, I print the info of all the running haproxy process in the console
and copy it below.

root 2367 0.0 0.0 4228 740 ? Ss 07:14 0:00
/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid

haproxy 2370 0.0 0.5 37692 5340 ? S 07:14 0:00
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

haproxy 2371 0.0 0.0 37692 924 ? Ss 07:14 0:00
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

root 2471 0.0 0.0 4228 636 ? Ss 07:14 0:00
/usr/sbin/haproxy-systemd-wrapper -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZ_IsG5HmdyY2LmdG3LSOco

nobody 2477 0.0 0.5 37676 5824 ? S 07:14 0:00
/usr/sbin/haproxy -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZ_IsG5HmdyY2LmdG3LSOco -Ds

nobody 2478 0.1 0.3 37676 3140 ? Ss 07:14 0:01
/usr/sbin/haproxy -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZ_IsG5HmdyY2LmdG3LSOco -Ds

ubuntu 2485 0.0 0.1 12916 1092 pts/0 S+ 07:36 0:00 grep
--color=auto haproxy

I run strace to trace the activities of all the above processes. When
sending request to the VIP, none of the above processes takes action to
receive the datagram. The details are omitted, since they give little
information.

Above all, I think the haproxy fail to receive the ingress traffic from the
IP and port it listens on.

What do you think? Look forward to your valuable comments. Thank you.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 9, 2017 in openstack-dev by Yipei_Niu (2,060 points)   2 8

10 Responses

0 votes

Hi Yipei,

I just tried to reproduce this and was not successful.

I setup a tenant network, added a web server to it, created a
loadbalancer VIP on the tenant network, added the webserver as a
member on the load balancer. I can curl from the tenant network
qdhcp- netns without issue.

Are you running a recent version of Octavia? This bug might be
impacting you: https://review.openstack.org/501915

Can you check if this routing table is present inside your amphora netns?

root@amphora-f58cdb7c-8fde-4eea-bd40-780664cfa49f:~# ip netns exec
amphora-haproxy ip route show table 1

default via dev eth1 onlink

It could be that since they are all on the same subnet there is a
return path issue in the kernel which this patch fixes with a policy
based route.

Michael

On Mon, Sep 25, 2017 at 1:33 AM, Yipei Niu newypei@gmail.com wrote:
Hi, all,

I encounter some problems when using Octavia. After installing octavia with
devstack, I create a load balancer named lb1 (VIP: 10.0.1.9, IP of VRRP
port: 10.0.1.3) for a subnet (10.0.1.0/24), then a listener, a pool, and two
members. All the resources are created successfully. The two members (VMs)
reside in the same subnet, whose IP are 10.0.1.6 and 10.0.1.7, respectively.
To simulate a web server in each VM, I run "while true; do echo -e "HTTP/1.0
200 OK\r\n\r\nWelcome to $VMIP" | sudo nc -l -p 80;done" to listen on port
80 and return the VM's IP if the request is accepted. I run "sudo ip netns
exec qdhcp-XXXX curl -v $VM
IP" to send requests to VMs, the "web servers"
in VMs work (already added corresponding security rules to the VMs). Then I
tried to run "sudo ip netns exec qdhcp-XXXX curl -v $VIP" to send requests,
the VMs do not respond, and finally returns a timeout error.

The configuration details in local.conf are as follows.

[[local|localrc]]

DATABASEPASSWORD=password
RABBIT
PASSWORD=password
SERVICEPASSWORD=password
SERVICE
TOKEN=password
ADMIN_PASSWORD=password

HOST_IP=192.168.56.9

LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOGCOLOR=True
SCREEN
LOGDIR=/opt/stack/logs

Neutron LBaaS

enableplugin neutron-lbaas https://github.com/openstack/neutron-lbaas.git
enable
plugin octavia https://github.com/openstack/octavia.git
ENABLEDSERVICES+=,q-lbaasv2
ENABLED
SERVICES+=,octavia,o-cw,o-hk,o-hm,o-api

disableservice horizon
disable
service tempest

To investigate the source of the error, I logon to the amphora. The details
of interfaces of amphora_haproxy network namespace are as follows.

1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast state
UP group default qlen 1000
link/ether fa:16:3e:42:bf:d9 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.3/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fe42:bfd9/64 scope link
validlft forever preferredlft forever

So I run "sudo ip netns exec amphora-haproxy tcpdump -i eth1 -nn 'tcp'" to
check whether amphora receive the request. The details are as follows.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C06:11:49.048973 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq
1717936594, win 28200, options [mss 1410,sackOK,TS val 28032309 ecr
0,nop,wscale 7], length 0
06:11:50.031976 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq 1717936594,
win 28200, options [mss 1410,sackOK,TS val 28032559 ecr 0,nop,wscale 7],
length 0
06:11:52.026565 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq 1717936594,
win 28200, options [mss 1410,sackOK,TS val 28033060 ecr 0,nop,wscale 7],
length 0
06:11:56.002577 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq 1717936594,
win 28200, options [mss 1410,sackOK,TS val 28034062 ecr 0,nop,wscale 7],
length 0
06:12:03.909721 IP 10.0.1.2.33700 > 10.0.1.9.80: Flags [S], seq 1717936594,
win 28200, options [mss 1410,sackOK,TS val 28036064 ecr 0,nop,wscale 7],
length 0

Based on the trace, we can see that amphora do receive the request, but
haproxy does not send handshake datagram to respond. Then, to see whether
haproxy in the amphora listens on the right IP and port, I print
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg in the
console and see the following info.

Configuration for lb1

global
daemon
user nobody
log /dev/log local0
log /dev/log local1 notice
stats socket /var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57.sock
mode 0666 level user

defaults
log global
retries 3
option redispatch
timeout connect 5000
timeout client 50000
timeout server 50000

frontend ec5ee7c5-7474-424f-9f44-71b338cf3e57
option httplog
bind 10.0.1.9:80
mode http
default_backend 40537c80-979d-49c9-b3ae-8504812c0f42

backend 40537c80-979d-49c9-b3ae-8504812c0f42
mode http
balance roundrobin
server 73dc9a1d-1e92-479b-a6f3-8debd0ea17b8 10.0.1.6:80 weight 1
server 4cdca33f-9cde-4ac2-a5bd-550d3e65f0f2 10.0.1.7:80 weight 1

Next, I print the info of all the running haproxy process in the console and
copy it below.

root 2367 0.0 0.0 4228 740 ? Ss 07:14 0:00
/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid
haproxy 2370 0.0 0.5 37692 5340 ? S 07:14 0:00
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 2371 0.0 0.0 37692 924 ? Ss 07:14 0:00
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 2471 0.0 0.0 4228 636 ? Ss 07:14 0:00
/usr/sbin/haproxy-systemd-wrapper -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZIsG5HmdyY2LmdG3LSOco
nobody 2477 0.0 0.5 37676 5824 ? S 07:14 0:00
/usr/sbin/haproxy -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZ
IsG5HmdyY2LmdG3LSOco -Ds
nobody 2478 0.1 0.3 37676 3140 ? Ss 07:14 0:01
/usr/sbin/haproxy -f
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/ec5ee7c5-7474-424f-9f44-71b338cf3e57/ec5ee7c5-7474-424f-9f44-71b338cf3e57.pid
-L A2GNEZ_IsG5HmdyY2LmdG3LSOco -Ds
ubuntu 2485 0.0 0.1 12916 1092 pts/0 S+ 07:36 0:00 grep
--color=auto haproxy

I run strace to trace the activities of all the above processes. When
sending request to the VIP, none of the above processes takes action to
receive the datagram. The details are omitted, since they give little
information.

Above all, I think the haproxy fail to receive the ingress traffic from the
IP and port it listens on.

What do you think? Look forward to your valuable comments. Thank you.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

Hi, Michael,

I think the octavia is the latest, since I pull the up-to-date repo of
octavia manually to my server before installation.

Anyway, I run "sudo ip netns exec amphora-haproxy ip route show table 1" in
the amphora, and find that the route table exists. The info is listed as
follows.

default via 10.0.1.1 dev eth1 onlink

I think it may not be the source.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 26, 2017 by Yipei_Niu (2,060 points)   2 8
0 votes

Hi, Michael,

The instructions are listed as follows.

First, create a net1.
$ neutron net-create net1
$ neutron subnet-create net1 10.0.1.0/24 --name subnet1

Second, boot two vms in net1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm2

Third, logon to the two vms, respectively. Here take vm1 as an example.
$ MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print
$1}')
$ while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo
nc -l -p 80 ; done&

Fourth, exit vms and update the default security group shared by the vms by
adding a rule of allowing traffic to port 80.
$ neutron security-group-rule-create --direction ingress --protocol tcp
--port-range-min 80 --port-range-max 80 --remote-ip-refix 0.0.0.0/0
$defaultsecuritygroup
Note: make sure "sudo ip netns exec $qdhcp-net1id curl -v $vmip" works.
In other words, make sure the vms can accept HTTP requests and return its
IP, respectively.

Fifth, create a lb, a listener, and a pool. Then add the two vms to the
pool as members.
$ neutron lbaas-loadbalancer-create --name lb1 subnet1
$ neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP
--protocol-port 80 --name listener1
$ neutron lbaas-pool-create --lb-algorithm ROUNDROBIN --listener listener1
--protocol HTTP --name pool1
$ neutron baas-member-create --subnet subnet1 --address $vm1
ip
--protocol-port 80 pool1
$ neutron baas-member-create --subnet subnet1 --address $vm2_ip
--protocol-port 80 pool1

Finally, try "sudo ip netns qdhcp-net1_id curl -v $VIP" to see whether
lbaas works.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 1:30 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I think the octavia is the latest, since I pull the up-to-date repo of
octavia manually to my server before installation.

Anyway, I run "sudo ip netns exec amphora-haproxy ip route show table 1"
in the amphora, and find that the route table exists. The info is listed as
follows.

default via 10.0.1.1 dev eth1 onlink

I think it may not be the source.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 27, 2017 by Yipei_Niu (2,060 points)   2 8
0 votes

Hi Yipei,

I ran this scenario today using octavia and had success. I'm not sure
what could be different.
I see you are using neutron-lbaas. I will build a devstack with
neutron-lbaas enabled and try that, but I can't think of what would
impact this test case by going through the neutron-lbaas path.

Michael

On Tue, Sep 26, 2017 at 7:27 PM, Yipei Niu newypei@gmail.com wrote:
Hi, Michael,

The instructions are listed as follows.

First, create a net1.
$ neutron net-create net1
$ neutron subnet-create net1 10.0.1.0/24 --name subnet1

Second, boot two vms in net1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm2

Third, logon to the two vms, respectively. Here take vm1 as an example.
$ MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print
$1}')
$ while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo nc
-l -p 80 ; done&

Fourth, exit vms and update the default security group shared by the vms by
adding a rule of allowing traffic to port 80.
$ neutron security-group-rule-create --direction ingress --protocol tcp
--port-range-min 80 --port-range-max 80 --remote-ip-refix 0.0.0.0/0
$defaultsecuritygroup
Note: make sure "sudo ip netns exec $qdhcp-net1id curl -v $vmip" works. In
other words, make sure the vms can accept HTTP requests and return its IP,
respectively.

Fifth, create a lb, a listener, and a pool. Then add the two vms to the pool
as members.
$ neutron lbaas-loadbalancer-create --name lb1 subnet1
$ neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP
--protocol-port 80 --name listener1
$ neutron lbaas-pool-create --lb-algorithm ROUNDROBIN --listener listener1
--protocol HTTP --name pool1
$ neutron baas-member-create --subnet subnet1 --address $vm1
ip
--protocol-port 80 pool1
$ neutron baas-member-create --subnet subnet1 --address $vm2_ip
--protocol-port 80 pool1

Finally, try "sudo ip netns qdhcp-net1_id curl -v $VIP" to see whether lbaas
works.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 1:30 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I think the octavia is the latest, since I pull the up-to-date repo of
octavia manually to my server before installation.

Anyway, I run "sudo ip netns exec amphora-haproxy ip route show table 1"
in the amphora, and find that the route table exists. The info is listed as
follows.

default via 10.0.1.1 dev eth1 onlink

I think it may not be the source.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

Hi, Michael,

Thanks a lot. Look forward to your further test. I try deploying a new
environment, too. Hope it can work well this time.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 10:27 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

The instructions are listed as follows.

First, create a net1.
$ neutron net-create net1
$ neutron subnet-create net1 10.0.1.0/24 --name subnet1

Second, boot two vms in net1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm2

Third, logon to the two vms, respectively. Here take vm1 as an example.
$ MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print
$1}')
$ while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo
nc -l -p 80 ; done&

Fourth, exit vms and update the default security group shared by the vms
by adding a rule of allowing traffic to port 80.
$ neutron security-group-rule-create --direction ingress --protocol tcp
--port-range-min 80 --port-range-max 80 --remote-ip-refix 0.0.0.0/0
$defaultsecuritygroup
Note: make sure "sudo ip netns exec $qdhcp-net1id curl -v $vmip" works.
In other words, make sure the vms can accept HTTP requests and return its
IP, respectively.

Fifth, create a lb, a listener, and a pool. Then add the two vms to the
pool as members.
$ neutron lbaas-loadbalancer-create --name lb1 subnet1
$ neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP
--protocol-port 80 --name listener1
$ neutron lbaas-pool-create --lb-algorithm ROUNDROBIN --listener
listener1 --protocol HTTP --name pool1
$ neutron baas-member-create --subnet subnet1 --address $vm1
ip
--protocol-port 80 pool1
$ neutron baas-member-create --subnet subnet1 --address $vm2_ip
--protocol-port 80 pool1

Finally, try "sudo ip netns qdhcp-net1_id curl -v $VIP" to see whether
lbaas works.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 1:30 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I think the octavia is the latest, since I pull the up-to-date repo of
octavia manually to my server before installation.

Anyway, I run "sudo ip netns exec amphora-haproxy ip route show table 1"
in the amphora, and find that the route table exists. The info is listed as
follows.

default via 10.0.1.1 dev eth1 onlink

I think it may not be the source.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Yipei_Niu (2,060 points)   2 8
0 votes

Hi Yipei,
Even running through neutron-lbaas I get the same successful test.

Just to double check, you are using the Octavia driver?

stack@devstackpy27-2:~$ sudo ip netns exec
qdhcp-4bcefe3e-038f-4a77-af4f-a560b6316a7a curl 172.21.1.16
Welcome to 172.21.1.17 connection 3

Michael

On Thu, Sep 28, 2017 at 7:46 AM, Yipei Niu newypei@gmail.com wrote:
Hi, Michael,

Thanks a lot. Look forward to your further test. I try deploying a new
environment, too. Hope it can work well this time.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 10:27 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

The instructions are listed as follows.

First, create a net1.
$ neutron net-create net1
$ neutron subnet-create net1 10.0.1.0/24 --name subnet1

Second, boot two vms in net1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm1
$ nova boot --flavor 1 --image $imageid --nic net-id=$net1id vm2

Third, logon to the two vms, respectively. Here take vm1 as an example.
$ MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print
$1}')
$ while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo
nc -l -p 80 ; done&

Fourth, exit vms and update the default security group shared by the vms
by adding a rule of allowing traffic to port 80.
$ neutron security-group-rule-create --direction ingress --protocol tcp
--port-range-min 80 --port-range-max 80 --remote-ip-refix 0.0.0.0/0
$defaultsecuritygroup
Note: make sure "sudo ip netns exec $qdhcp-net1id curl -v $vmip" works.
In other words, make sure the vms can accept HTTP requests and return its
IP, respectively.

Fifth, create a lb, a listener, and a pool. Then add the two vms to the
pool as members.
$ neutron lbaas-loadbalancer-create --name lb1 subnet1
$ neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP
--protocol-port 80 --name listener1
$ neutron lbaas-pool-create --lb-algorithm ROUNDROBIN --listener
listener1 --protocol HTTP --name pool1
$ neutron baas-member-create --subnet subnet1 --address $vm1
ip
--protocol-port 80 pool1
$ neutron baas-member-create --subnet subnet1 --address $vm2_ip
--protocol-port 80 pool1

Finally, try "sudo ip netns qdhcp-net1_id curl -v $VIP" to see whether
lbaas works.

Best regards,
Yipei

On Wed, Sep 27, 2017 at 1:30 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I think the octavia is the latest, since I pull the up-to-date repo of
octavia manually to my server before installation.

Anyway, I run "sudo ip netns exec amphora-haproxy ip route show table 1"
in the amphora, and find that the route table exists. The info is listed as
follows.

default via 10.0.1.1 dev eth1 onlink

I think it may not be the source.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

Hi, Michael,

I checked neutronlbaas.conf, the service provider is set as “LOADBALANCERV2:Octavia:neutronlbaas.drivers.octavia.driver.OctaviaDriver:default”.

Further, I installed a new environment in a new VM today, it still doesn’t work.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 8, 2017 by Yipei_Niu (2,060 points)   2 8
0 votes

Hi, Michael,

I once installed a successful environment, so I move the amphora image of the environment to my latest environment which doesn’t work. I create a load balancer in the latest environment using the old amphora image, and it works.

So I think there is some problem with the amphora image of my latest environment. The detailed info of my amphora image is as follows.

stack@VM-IP10:~$ glance image-show 3bc4a09e-86b5-4e54-b468-b3d9c7ac8607
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 4bd41781226068fe91bf181efd2df8c3 |
| containerformat | bare |
| created
at | 2017-09-28T02:50:48Z |
| diskformat | qcow2 |
| id | 3bc4a09e-86b5-4e54-b468-b3d9c7ac8607 |
| min
disk | 0 |
| minram | 0 |
| name | amphora-x64-haproxy |
| owner | 01fd6fdee6ba436abe1a4bfe5defeea2 |
| protected | False |
| size | 678293504 |
| status | active |
| tags | ["amphora"] |
| updated
at | 2017-09-28T02:51:13Z |
| virtual_size | None |
| visibility | public |
+------------------+———————————————————+

Is the checksum the same to yours?

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 8, 2017 by Yipei_Niu (2,060 points)   2 8
0 votes

When was the old image built and when was the new one built? I noticed my
images had stopped working on our older (Liberty) production cloud, and
discovered that recently Ubuntu and Centos have both upgraded cloud-init to
0.7.9 in their base images (from something like 0.7.5), which finished the
deprecation of the old cloud-init config format that our older cloud
injects. If they're booting in Nova but not allowing connections, see if
you can console in or inspect the logs. Cloud-init was not bringing up the
networking properly on the old version for me.

On Sun, Oct 8, 2017, 02:59 Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I once installed a successful environment, so I move the amphora image of
the environment to my latest environment which doesn’t work. I create a
load balancer in the latest environment using the old amphora image, and it
works.

So I think there is some problem with the amphora image of my latest
environment. The detailed info of my amphora image is as follows.

stack@VM-IP10:~$ glance image-show 3bc4a09e-86b5-4e54-b468-b3d9c7ac8607
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 4bd41781226068fe91bf181efd2df8c3 |
| containerformat | bare |
| created
at | 2017-09-28T02:50:48Z |
| diskformat | qcow2 |
| id | 3bc4a09e-86b5-4e54-b468-b3d9c7ac8607 |
| min
disk | 0 |
| minram | 0 |
| name | amphora-x64-haproxy |
| owner | 01fd6fdee6ba436abe1a4bfe5defeea2 |
| protected | False |
| size | 678293504 |
| status | active |
| tags | ["amphora"] |
| updated
at | 2017-09-28T02:51:13Z |
| virtual_size | None |
| visibility | public |
+------------------+———————————————————+

Is the checksum the same to yours?

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 9, 2017 by flux.adam_at_gmail.c (1,020 points)  
0 votes

Hi, Adam,

The following details of my older amphora image are as follows.

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | aa279b9ae1f265a232266e17ea04065c |
| containerformat | bare |
| created
at | 2017-07-24T10:46:34Z |
| diskformat | qcow2 |
| id | 5d6d5f69-a4b1-440a-b6bb-10bf945e3f85 |
| min
disk | 0 |
| minram | 0 |
| name | amphora-x64-haproxy |
| owner | c4f8c2a8fadc4b7c915f299db95c88a8 |
| protected | False |
| size | 674900480 |
| status | active |
| tags | ["amphora"] |
| updated
at | 2017-07-24T10:47:54Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+

It is created on 2017-07-24, while my latest one is created at 2017-09-28.

I installed octavia with devstack, but it doesn't work. I discussed the
issue with Michael, he cannot reproduce the error. Please refer to the
history mails here
http://lists.openstack.org/pipermail/openstack-dev/2017-September/122814.html.
I can console in the amphora and trace the packet is received by my
amphora, but haproxy fails to respond. More details are here
http://lists.openstack.org/pipermail/openstack-dev/2017-September/122587.html.
I think maybe there is something wrong with my latest amphora image, I
hence use an older amphora image in the latest environment, it works. As a
result, the networking are supposed to be fine. What do you think?

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 9, 2017 by Yipei_Niu (2,060 points)   2 8
...