settingsLogin | Registersettings

[openstack-dev] [octavia] amphora fails to send request to members

0 votes

Hi, all,

I created a lb whose vip is 10.0.1.4. When requesting the vip, i cannot
receive the responses. Hence, I console in the amphora and trace packets
handled by eth0 in the amphora-haproxy network namespace. The detailed info
is as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30692853 ecr 0,nop,wscale 7],
length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30693359 ecr 0,nop,wscale 7],
length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

14 packets captured
14 packets received by filter
0 packets dropped by kernel

As shown above, the amphora can receive the requests comes from outside but
it fails to find 10.0.1.1. As a result, amphora cannot send the request to
its load balancing member.

To further demonstrate my environment, the route table and arp cache are
listed as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy route
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
default 10.0.1.1 0.0.0.0 UG 0 0 0 eth1
10.0.1.0 * 255.255.255.0 U 0 0 0 eth1

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy arp
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Address HWtype HWaddress Flags Mask
Iface
10.0.1.2 ether fa:16:3e:62:68:a8 C
eth1
10.0.1.1 (incomplete)
eth1

What do you think? Look forward to your comments. Thank you.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 14, 2017 in openstack-dev by Yipei_Niu (2,060 points)   4

11 Responses

0 votes

Hi Yipei,

I need some more information to help you out. Can you provide the following?

  1. What version of Octavia you are using.
  2. "openstack server list" output for the amphora.
  3. "openstack loadbalancer show" for the load balancer.
  4. "openstack loadbalancer listener show" for the listener.
  5. "openstack loadbalancer member list" for the load balancer.
  6. "openstack subnet list" showing all of the subnets used by the
    lb-mgmt-net, VIP, and the members.
  7. "openstack subnet show" for each of the lb-mgmt-net, VIP, and member subnets
  8. "ifconfig" from inside the amphora and from inside the netns on the amphora
  9. curl command you are using to connect to the VIP
  10. "netstat -rn" from the host you are running curl from
  11. Inside the amphora, cd /opt/amphora-agent, "git log" cut/paste the
    top two commit messages.

That should help us see what your configuration is and narrow down
what is going on.
As always, you can also try to have an interactive session with us in

openstack-lbaas on IRC. However, with the summit the hours folks are

available to answer questions may be a bit limited this week.

Michael

On Tue, Nov 7, 2017 at 11:32 PM, Yipei Niu newypei@gmail.com wrote:
Hi, all,

I created a lb whose vip is 10.0.1.4. When requesting the vip, i cannot
receive the responses. Hence, I console in the amphora and trace packets
handled by eth0 in the amphora-haproxy network namespace. The detailed info
is as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30692853 ecr 0,nop,wscale 7],
length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30693359 ecr 0,nop,wscale 7],
length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

14 packets captured
14 packets received by filter
0 packets dropped by kernel

As shown above, the amphora can receive the requests comes from outside but
it fails to find 10.0.1.1. As a result, amphora cannot send the request to
its load balancing member.

To further demonstrate my environment, the route table and arp cache are
listed as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy route
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
default 10.0.1.1 0.0.0.0 UG 0 0 0 eth1
10.0.1.0 * 255.255.255.0 U 0 0 0 eth1

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy arp
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Address HWtype HWaddress Flags Mask
Iface
10.0.1.2 ether fa:16:3e:62:68:a8 C
eth1
10.0.1.1 (incomplete)
eth1

What do you think? Look forward to your comments. Thank you.

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Michael_Johnson (4,380 points)   3 5
0 votes

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 |
    amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
    | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4;
    net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start": "10.0.1.1",
    "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally return
    a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 9, 2017 by Yipei_Niu (2,060 points)   4
0 votes

Hi Yipei,

I see a few things that are odd:
stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

This means that the connection is not working between curl and the
HAProxy process. If HAProxy was just unable to reach the member you
would get a 503 error.

The other thing that does not make sense is the routing table on the
amphora-haproxy namespace.
It has a route of:
default 10.0.1.1 0.0.0.0 UG 0 0 0 eth1

But your subnet has a gateway:
|gateway_ip | 10.0.1.10 |

Are there multiple routers on the VIP network? Or multiple DHCP
services? Since neutron is configured for DHCP on that subnet the
amphora will get it's default gateway from the DHCP packet it uses to
configure the base interface eth1.

Michael

On Wed, Nov 8, 2017 at 10:50 PM, Yipei Niu newypei@gmail.com wrote:
Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 |
    amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f | ACTIVE | - | Running
    | lb-mgmt-net1=192.168.1.4; net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  4. The info of the listener is as follows.
    +---------------------------+------------------------------------------------+
    | Field | Value
    |
    +---------------------------+------------------------------------------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+------------------------------------------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid |
    admin
    state_up |
    +--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two subnets
    are listed as follows.
    +--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools |
    +--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start": "10.0.1.1",
    "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end": "10.0.1.254"} |
    +--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally return a
    timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2017 by Michael_Johnson (4,380 points)   3 5
0 votes

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30692853 ecr 0,nop,wscale 7],
length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30693359 ecr 0,nop,wscale 7],
length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

*So if the subnet is not attached to any router, why does haproxy try to
find the gateway ip that does not exist at all? Maybe that is the reason
why haproxy receives the packet from curl but fail to respond. *

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle (https://wiki.openstack.org/wiki/Tricircle)
are installed together. Because of the cross-neutron mechanism of
tricircle, the gateway ip of subnet in that region is 10.0.1.10. But I can
make sure that gateway ip (10.0.1.1 or 10.0.1.10) does not exist in the
network, since there is no router at all. This error also happens in my
another environment where octavia is installed alone. The environment is
installed on Oct. 6, and all the repos are the latest at that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message
    is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 | amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
    | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4;
    net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start": "10.0.1.1",
    "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally return
    a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2017 by Yipei_Niu (2,060 points)   4
0 votes

The actual gateway address does not matter to Octavia/amphora. It
gets that value from DHCP or from neutron if a static address was
assigned from neutron.
My concern is that the subnet gateway 10.0.1.10 does not match the
gateway address DHCP gave us 10.0.1.1.
Technically, since the two addresses are on the same IP subnet the
router should not matter, it was just an observation that I noticed as
something is not right with neutron.
From inside the amphora-haproxy namespace, can you curl and/or ping to 10.0.1.3?
From inside the qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 namespace
on the host, can you curl/ping your member 10.0.1.3?

I am thinking that the host qdhcp-310fea4b-36ae-4617-b499-5936e8eda842
namespace does not have connectivity to to the nova instances somehow.

Michael

On Fri, Nov 10, 2017 at 1:24 AM, Yipei Niu newypei@gmail.com wrote:
Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30692853 ecr 0,nop,wscale 7],
length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601,
win 28200, options [mss 1410,sackOK,TS val 30693359 ecr 0,nop,wscale 7],
length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

So if the subnet is not attached to any router, why does haproxy try to find
the gateway ip that does not exist at all? Maybe that is the reason why
haproxy receives the packet from curl but fail to respond.

I think the gateway ip (10.0.1.10) confuses you. Actually, in my environment
octavia and tricircle (https://wiki.openstack.org/wiki/Tricircle) are
installed together. Because of the cross-neutron mechanism of tricircle, the
gateway ip of subnet in that region is 10.0.1.10. But I can make sure that
gateway ip (10.0.1.1 or 10.0.1.10) does not exist in the network, since
there is no router at all. This error also happens in my another environment
where octavia is installed alone. The environment is installed on Oct. 6,
and all the repos are the latest at that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message
    is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
| 33bd02cb-f853-404d-a705-99bc1b04a178 |
amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f | ACTIVE | - | Running
| lb-mgmt-net1=192.168.1.4; net1=10.0.1.8 |
| dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
| ACTIVE | - | Running | net1=10.0.1.3
|
| 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
| ACTIVE | - | Running | net4=10.0.4.3
|

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+

  1. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  2. The info of the listener is as follows.

+---------------------------+------------------------------------------------+
| Field | Value
|

+---------------------------+------------------------------------------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | d0042605-da50-4048-b298-660420b0a1d2
|
| default
tlscontainerref |
|
| description |
|
| id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
|
| loadbalancers | {"id":
"51cba1d5-cc3c-48ff-b41e-839619093334"} |
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | c2a97a04cb6d4f25bdcb8b3f263c869e
|

+---------------------------+------------------------------------------------+

  1. The members of the load balancer lb1 are as follows.

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| id | name | tenantid
| address | protocol
port | weight | subnetid |
admin
state_up |

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| 420c905c-1077-46c9-8b04-526a59d93376 | |
c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+

  1. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
| id | name | tenantid
| cidr | allocation
pools |

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
| 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
"192.168.1.10", "end": "192.168.1.254"} |
| | |
| | {"start": "192.168.1.1", "end": "192.168.1.8"} |
| cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start": "10.0.1.1",
"end": "10.0.1.9"} |
| | |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+

  1. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally return
    a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2017 by Michael_Johnson (4,380 points)   3 5
0 votes

Hi, Michael,

I tried to run to command, and I think the amphora can connect to the
member (10.0.1.3). The results are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ping 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmpseq=1 ttl=64 time=189 ms
64 bytes from 10.0.1.3: icmp
seq=2 ttl=64 time=1.72 ms
^C
--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.722/95.855/189.989/94.134 ms

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Welcome to 10.0.1.3

stack@devstack-1:~$ sudo ip netns exec
qdhcp-310fea4b-36ae-4617-b499-5936e8eda842
curl 10.0.1.3
Welcome to 10.0.1.3

As mentioned in my previous mail, I also have an environment installed with
octavia alone, where the error reproduces. In that environment, I also
tried the above commands, and have the same results. The member can be
reached from host and amphora.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.5
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
Welcome to 10.0.1.5

stack@stack-VirtualBox:~$ sudo ip netns exec
qdhcp-13185eec-0996-4a08-b353-6775d5926b4c
curl 10.0.1.5
Welcome to 10.0.1.5

In this environment, haproxy also tries to find the gateway ip 10.0.1.1.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C08:58:53.997948 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330593 ecr
0,nop,wscale 7], length 0
08:58:54.011771 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:54.991600 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:55.018542 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910330850 ecr 0,nop,wscale 7],
length 0
08:58:55.991703 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.015187 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.034507 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910331354 ecr 0,nop,wscale 7],
length 0
08:58:58.016438 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.017650 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.115960 ARP, Request who-has 10.0.1.9 tell 10.0.1.2, length 28
08:58:59.116293 ARP, Reply 10.0.1.9 is-at fa:16:3e:dc:9e:61, length 28
08:59:01.031434 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:01.162845 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910332386 ecr 0,nop,wscale 7],
length 0
08:59:02.031095 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:03.035527 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

15 packets captured
15 packets received by filter
0 packets dropped by kernel

And the gateway ip requested by haproxy is the same as it in the subnet.
+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

Some other info in this octavia env is as follows.

The info of load balancer:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| adminstateup | True |
| description | |
| id | d087d3b4-afbe-4af6-8b31-5e86fc97da1b |
| listeners | {"id": "d22644d2-cc40-44f1-b37f-2bb4f555f9b9"} |
| name | lb1 |
| operatingstatus | ONLINE |
| pools | {"id": "bc95c8e0-8475-4d97-9606-76c431e78ef7"} |
| provider | octavia |
| provisioning
status | ACTIVE |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| vip
address | 10.0.1.9 |
| vipportid | 902a78e7-a618-455b-91c7-cd36595475cc |
| vipsubnetid | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
+---------------------+------------------------------------------------+

The info of VMs:
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| 50ae3581-94fc-43ce-b53c-728715cd5597 |
amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
| ACTIVE | - | Running | lb-mgmt-net=192.168.0.10;
net1=10.0.1.11 |
| d19b565d-14aa-4679-9d98-ff51461cd625 | vm1
| ACTIVE | - | Running | net1=10.0.1.5
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+

The info of listener:
+---------------------------+-------------------------------
-----------------+
| Field | Value
|
+---------------------------+-------------------------------
-----------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | bc95c8e0-8475-4d97-9606-76c431e78ef7
|
| default
tlscontainerref |
|
| description |
|
| id | d22644d2-cc40-44f1-b37f-2bb4f555f9b9
|
| loadbalancers | {"id": "d087d3b4-afbe-4af6-8b31-5e86fc97da1b"}
|
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | e59bb8f3bf9342aba02f9ba5804ed2fb
|
+---------------------------+-------------------------------
-----------------+

So I think the amphora can reach the member, it just cannot respond to
curl, hence failing to balance load across members. Maybe there is
something wrong with the image of the amphora. Since I replaced the latest
amphora image with an old one (built on 2017-07-24), it works. Totally same
environment except the amphora image.

Best regards,
Yipei

On Fri, Nov 10, 2017 at 5:24 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr
0,nop,wscale 7], length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr
0,nop,wscale 7], length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

*So if the subnet is not attached to any router, why does haproxy try to
find the gateway ip that does not exist at all? Maybe that is the reason
why haproxy receives the packet from curl but fail to respond. *

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle (https://wiki.openstack.org/
wiki/Tricircle) are installed together. Because of the cross-neutron
mechanism of tricircle, the gateway ip of subnet in that region is
10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or 10.0.1.10) does
not exist in the network, since there is no router at all. This error also
happens in my another environment where octavia is installed alone. The
environment is installed on Oct. 6, and all the repos are the latest at
that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message
    is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 | amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
    | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4;
    net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start":
    "10.0.1.1", "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally
    return a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 11, 2017 by Yipei_Niu (2,060 points)   4
0 votes

Yipei,

I am struggling to follow some of the details as I see different information:

+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

and

+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

That said, the comment about the version of the amphora is
interesting. We did this patch:
https://review.openstack.org/#/c/501915/ that may be playing a role
here.
It should just be adding a default route for traffic originating from
the VIP, which would not impact local subnet traffic, but it is the
only change I can think of that might be related.

Can you send the output of:
sudo ip netns exec amphora-haproxy ip route show table all

and

sudo ip netns exec amphora-haproxy ip rule show

from the failing amphora?

Michael

On Fri, Nov 10, 2017 at 7:24 PM, Yipei Niu newypei@gmail.com wrote:
Hi, Michael,

I tried to run to command, and I think the amphora can connect to the member
(10.0.1.3). The results are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ping 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmpseq=1 ttl=64 time=189 ms
64 bytes from 10.0.1.3: icmp
seq=2 ttl=64 time=1.72 ms
^C
--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.722/95.855/189.989/94.134 ms

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Welcome to 10.0.1.3

stack@devstack-1:~$ sudo ip netns exec
qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.3
Welcome to 10.0.1.3

As mentioned in my previous mail, I also have an environment installed with
octavia alone, where the error reproduces. In that environment, I also tried
the above commands, and have the same results. The member can be reached
from host and amphora.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.5
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
Welcome to 10.0.1.5

stack@stack-VirtualBox:~$ sudo ip netns exec
qdhcp-13185eec-0996-4a08-b353-6775d5926b4c curl 10.0.1.5
Welcome to 10.0.1.5

In this environment, haproxy also tries to find the gateway ip 10.0.1.1.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C08:58:53.997948 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330593 ecr
0,nop,wscale 7], length 0
08:58:54.011771 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:54.991600 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:55.018542 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910330850 ecr 0,nop,wscale 7],
length 0
08:58:55.991703 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.015187 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.034507 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910331354 ecr 0,nop,wscale 7],
length 0
08:58:58.016438 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.017650 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.115960 ARP, Request who-has 10.0.1.9 tell 10.0.1.2, length 28
08:58:59.116293 ARP, Reply 10.0.1.9 is-at fa:16:3e:dc:9e:61, length 28
08:59:01.031434 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:01.162845 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq 4080898035,
win 28200, options [mss 1410,sackOK,TS val 3910332386 ecr 0,nop,wscale 7],
length 0
08:59:02.031095 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:03.035527 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

15 packets captured
15 packets received by filter
0 packets dropped by kernel

And the gateway ip requested by haproxy is the same as it in the subnet.
+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

Some other info in this octavia env is as follows.

The info of load balancer:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| adminstateup | True |
| description | |
| id | d087d3b4-afbe-4af6-8b31-5e86fc97da1b |
| listeners | {"id": "d22644d2-cc40-44f1-b37f-2bb4f555f9b9"} |
| name | lb1 |
| operatingstatus | ONLINE |
| pools | {"id": "bc95c8e0-8475-4d97-9606-76c431e78ef7"} |
| provider | octavia |
| provisioning
status | ACTIVE |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| vip
address | 10.0.1.9 |
| vipportid | 902a78e7-a618-455b-91c7-cd36595475cc |
| vipsubnetid | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
+---------------------+------------------------------------------------+

The info of VMs:
+--------------------------------------+----------------------------------------------+--------+------------+-------------+------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|
+--------------------------------------+----------------------------------------------+--------+------------+-------------+------------------------------------------+
| 50ae3581-94fc-43ce-b53c-728715cd5597 |
amphora-dcbff58a-d418-4271-9374-9de2fd063ce9 | ACTIVE | - | Running
| lb-mgmt-net=192.168.0.10; net1=10.0.1.11 |
| d19b565d-14aa-4679-9d98-ff51461cd625 | vm1
| ACTIVE | - | Running | net1=10.0.1.5
|
+--------------------------------------+----------------------------------------------+--------+------------+-------------+------------------------------------------+

The info of listener:
+---------------------------+------------------------------------------------+
| Field | Value
|
+---------------------------+------------------------------------------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | bc95c8e0-8475-4d97-9606-76c431e78ef7
|
| default
tlscontainerref |
|
| description |
|
| id | d22644d2-cc40-44f1-b37f-2bb4f555f9b9
|
| loadbalancers | {"id": "d087d3b4-afbe-4af6-8b31-5e86fc97da1b"}
|
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | e59bb8f3bf9342aba02f9ba5804ed2fb
|
+---------------------------+------------------------------------------------+

So I think the amphora can reach the member, it just cannot respond to curl,
hence failing to balance load across members. Maybe there is something wrong
with the image of the amphora. Since I replaced the latest amphora image
with an old one (built on 2017-07-24), it works. Totally same environment
except the amphora image.

Best regards,
Yipei

On Fri, Nov 10, 2017 at 5:24 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr
0,nop,wscale 7], length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr
0,nop,wscale 7], length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

So if the subnet is not attached to any router, why does haproxy try to
find the gateway ip that does not exist at all? Maybe that is the reason why
haproxy receives the packet from curl but fail to respond.

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle
(https://wiki.openstack.org/wiki/Tricircle) are installed together. Because
of the cross-neutron mechanism of tricircle, the gateway ip of subnet in
that region is 10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or
10.0.1.10) does not exist in the network, since there is no router at all.
This error also happens in my another environment where octavia is installed
alone. The environment is installed on Oct. 6, and all the repos are the
latest at that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit message
    is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+
| 33bd02cb-f853-404d-a705-99bc1b04a178 |
amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f | ACTIVE | - | Running
| lb-mgmt-net1=192.168.1.4; net1=10.0.1.8 |
| dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
| ACTIVE | - | Running | net1=10.0.1.3
|
| 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
| ACTIVE | - | Running | net4=10.0.4.3
|

+--------------------------------------+----------------------------------------------+--------+------------+-------------+---------------------------------------------------------+

  1. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  2. The info of the listener is as follows.

+---------------------------+------------------------------------------------+
| Field | Value
|

+---------------------------+------------------------------------------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | d0042605-da50-4048-b298-660420b0a1d2
|
| default
tlscontainerref |
|
| description |
|
| id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
|
| loadbalancers | {"id":
"51cba1d5-cc3c-48ff-b41e-839619093334"} |
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | c2a97a04cb6d4f25bdcb8b3f263c869e
|

+---------------------------+------------------------------------------------+

  1. The members of the load balancer lb1 are as follows.

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| id | name | tenantid
| address | protocol
port | weight | subnetid |
admin
state_up |

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+
| 420c905c-1077-46c9-8b04-526a59d93376 | |
c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |

+--------------------------------------+------+----------------------------------+----------+---------------+--------+--------------------------------------+----------------+

  1. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
| id | name | tenantid
| cidr | allocation
pools |

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+
| 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
"192.168.1.10", "end": "192.168.1.254"} |
| | |
| | {"start": "192.168.1.1", "end": "192.168.1.8"} |
| cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start": "10.0.1.1",
"end": "10.0.1.9"} |
| | |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |

+--------------------------------------+-----------------+----------------------------------+----------------+---------------------------------------------------+

  1. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------------------+
    | Field | Value |
    +-------------------+---------------------------------------------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24 |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1 |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally
    return a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 13, 2017 by Michael_Johnson (4,380 points)   3 5
0 votes

Hi, Michael,

Thanks a lot for your comments.

In the environment where octavia and tricircle are installed together, I
created a router and attached subnet1 to it. Then I bind the mac address of
10.0.1.9 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
manually making amphora knows the mac address of 10.0.1.1 (actually it is
the mac of 10.0.1.9, since 10.0.1.1 does not exist), it works.

To make the situation clear, I run the commands in the environment
installed octavia alone. Please note that in this environment, there is no
router. The results are as follows.

stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--------------------------------------+------+----------------------------------+-------------+---------------------+----------+
| id | name | tenantid
| vip
address | provisioning_status | provider |
+--------------------------------------+------+----------------------------------+-------------+---------------------+----------+
| d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1 |
e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9 | ACTIVE |
octavia |
+--------------------------------------+------+----------------------------------+-------------+---------------------+----------+

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip addr
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast state
UP group default qlen 1000
link/ether fa:16:3e:dc:9e:61 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.11/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fedc:9e61/64 scope link
validlft forever preferredlft forever

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.1 dev eth1 onlink
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.11
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.11
local 10.0.1.9 dev eth1 table local proto kernel scope host src
10.0.1.11
local 10.0.1.11 dev eth1 table local proto kernel scope host src
10.0.1.11
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.11
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
error -101 pref medium
local fe80::f816:3eff:fedc:9e61 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
error -101 pref medium

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
0: from all lookup local
100: from 10.0.1.9 lookup 1
32766: from all lookup main
32767: from all lookup default

If there is no router, the packets are supposed to forwarded in layer 2, as
amphora is plugged with ports of every member's subnet. The priority of
“from 10.0.1.9 lookup 1” is higher than that of "from all lookup local".
Maybe this patch (https://review.openstack.org/#/c/501915/) affects the l2
traffic.

Best regards,
Yipei

On Sat, Nov 11, 2017 at 11:24 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I tried to run to command, and I think the amphora can connect to the
member (10.0.1.3). The results are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ping 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmpseq=1 ttl=64 time=189 ms
64 bytes from 10.0.1.3: icmp
seq=2 ttl=64 time=1.72 ms
^C
--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.722/95.855/189.989/94.134 ms

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Welcome to 10.0.1.3

stack@devstack-1:~$ sudo ip netns exec qdhcp-310fea4b-36ae-4617-b499-5936e8eda842
curl 10.0.1.3
Welcome to 10.0.1.3

As mentioned in my previous mail, I also have an environment installed
with octavia alone, where the error reproduces. In that environment, I also
tried the above commands, and have the same results. The member can be
reached from host and amphora.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy curl 10.0.1.5
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
Welcome to 10.0.1.5

stack@stack-VirtualBox:~$ sudo ip netns exec
qdhcp-13185eec-0996-4a08-b353-6775d5926b4c curl 10.0.1.5
Welcome to 10.0.1.5

In this environment, haproxy also tries to find the gateway ip 10.0.1.1.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C08:58:53.997948 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330593 ecr
0,nop,wscale 7], length 0
08:58:54.011771 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:54.991600 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:55.018542 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330850 ecr
0,nop,wscale 7], length 0
08:58:55.991703 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.015187 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.034507 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910331354 ecr
0,nop,wscale 7], length 0
08:58:58.016438 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.017650 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.115960 ARP, Request who-has 10.0.1.9 tell 10.0.1.2, length 28
08:58:59.116293 ARP, Reply 10.0.1.9 is-at fa:16:3e:dc:9e:61, length 28
08:59:01.031434 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:01.162845 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910332386 ecr
0,nop,wscale 7], length 0
08:59:02.031095 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:03.035527 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

15 packets captured
15 packets received by filter
0 packets dropped by kernel

And the gateway ip requested by haproxy is the same as it in the subnet.
+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

Some other info in this octavia env is as follows.

The info of load balancer:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| adminstateup | True |
| description | |
| id | d087d3b4-afbe-4af6-8b31-5e86fc97da1b |
| listeners | {"id": "d22644d2-cc40-44f1-b37f-2bb4f555f9b9"} |
| name | lb1 |
| operatingstatus | ONLINE |
| pools | {"id": "bc95c8e0-8475-4d97-9606-76c431e78ef7"} |
| provider | octavia |
| provisioning
status | ACTIVE |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| vip
address | 10.0.1.9 |
| vipportid | 902a78e7-a618-455b-91c7-cd36595475cc |
| vipsubnetid | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
+---------------------+------------------------------------------------+

The info of VMs:
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| 50ae3581-94fc-43ce-b53c-728715cd5597 | amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
| ACTIVE | - | Running | lb-mgmt-net=192.168.0.10;
net1=10.0.1.11 |
| d19b565d-14aa-4679-9d98-ff51461cd625 | vm1
| ACTIVE | - | Running | net1=10.0.1.5
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+

The info of listener:
+---------------------------+-------------------------------
-----------------+
| Field | Value
|
+---------------------------+-------------------------------
-----------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | bc95c8e0-8475-4d97-9606-76c431e78ef7
|
| default
tlscontainerref |
|
| description |
|
| id | d22644d2-cc40-44f1-b37f-2bb4f555f9b9
|
| loadbalancers | {"id": "d087d3b4-afbe-4af6-8b31-5e86fc97da1b"}
|
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | e59bb8f3bf9342aba02f9ba5804ed2fb
|
+---------------------------+-------------------------------
-----------------+

So I think the amphora can reach the member, it just cannot respond to
curl, hence failing to balance load across members. Maybe there is
something wrong with the image of the amphora. Since I replaced the latest
amphora image with an old one (built on 2017-07-24), it works. Totally
same environment except the amphora image.

Best regards,
Yipei

On Fri, Nov 10, 2017 at 5:24 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr
0,nop,wscale 7], length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr
0,nop,wscale 7], length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

*So if the subnet is not attached to any router, why does haproxy try to
find the gateway ip that does not exist at all? Maybe that is the reason
why haproxy receives the packet from curl but fail to respond. *

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle (https://wiki.openstack.org/wi
ki/Tricircle) are installed together. Because of the cross-neutron
mechanism of tricircle, the gateway ip of subnet in that region is
10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or 10.0.1.10) does
not exist in the network, since there is no router at all. This error also
happens in my another environment where octavia is installed alone. The
environment is installed on Oct. 6, and all the repos are the latest at
that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit
    message is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 | amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
    | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4;
    net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+------------------------------------------------+
    | Field | Value |
    +---------------------+------------------------------------------------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334 |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"} |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"} |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
    +---------------------+------------------------------------------------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start":
    "10.0.1.1", "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows,
    respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------
    ------------+
    | Field | Value |
    +-------------------+---------------------------------------
    ------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"} |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"} |
    | cidr | 192.168.1.0/24
    |
    | created
    at | 2017-11-05T12:14:45Z |
    | description | |
    | dnsnameservers | |
    | enable
    dhcp | True |
    | gatewayip | 192.168.1.9 |
    | host
    routes | |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1
    |
    | ipversion | 4 |
    | ipv6
    addressmode | |
    | ipv6
    ramode | |
    | name | lb-mgmt-subnet1 |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea
    |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | revision
    number | 0 |
    | servicetypes | |
    | subnetpool
    id | |
    | tags | |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | updated
    at | 2017-11-05T12:14:45Z |
    +-------------------+---------------------------------------
    ------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally
    return a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt
    Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0
    o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0
    enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
    enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0
    o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0
    enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
    virbr0

  3. In the amphora, the first two commit message of amphora-agent are as
    follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Yipei_Niu (2,060 points)   4
0 votes

Hi, Michael,

Sorry about the typo in the last mail. Please just ignore the last mail.

In the environment where octavia and tricircle are installed together, I
created a router and attached subnet1 to it. Then I bind the mac address of
10.0.1.10 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
manually making amphora knows the mac address of 10.0.1.1 (actually it is
the mac of 10.0.1.10, since 10.0.1.1 does not exist), it works.

I also run the commands in this environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.10 dev eth1
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.8
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.8
local 10.0.1.4 dev eth1 table local proto kernel scope host src
10.0.1.8
local 10.0.1.8 dev eth1 table local proto kernel scope host src
10.0.1.8
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.8
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
error -101 pref medium
local fe80::f816:3eff:febe:5ad5 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
error -101 pref medium
ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
0: from all lookup local
100: from 10.0.1.4 lookup 1
32766: from all lookup main
32767: from all lookup default

To make the situation clear, I run the commands in the environment
installed octavia alone. Please note that in this environment, there is no
router. The results are as follows.

stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| id | name | tenantid
| vip
address | provisioning_status | provider |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1 |
e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9 | ACTIVE |
octavia |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip addr
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast state
UP group default qlen 1000
link/ether fa:16:3e:dc:9e:61 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.11/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fedc:9e61/64 scope link
validlft forever preferredlft forever

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.1 dev eth1 onlink
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.11
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.11
local 10.0.1.9 dev eth1 table local proto kernel scope host src
10.0.1.11
local 10.0.1.11 dev eth1 table local proto kernel scope host src
10.0.1.11
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.11
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:fedc:9e61 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
0: from all lookup local
100: from 10.0.1.9 lookup 1
32766: from all lookup main
32767: from all lookup default

If there is no router, the packets are supposed to forwarded in layer 2, as
amphora is plugged with ports of every member's subnet. The priority of
“from 10.0.1.9 lookup 1” is higher than that of "from all lookup local".
Maybe this patch (https://review.openstack.org/#/c/501915/) affects the l2
traffic.

Best regards,
Yipei

On Tue, Nov 14, 2017 at 10:40 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your comments.

In the environment where octavia and tricircle are installed together, I
created a router and attached subnet1 to it. Then I bind the mac address of
10.0.1.9 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
manually making amphora knows the mac address of 10.0.1.1 (actually it is
the mac of 10.0.1.9, since 10.0.1.1 does not exist), it works.

To make the situation clear, I run the commands in the environment
installed octavia alone. Please note that in this environment, there is no
router. The results are as follows.

stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| id | name | tenantid
| vip
address | provisioning_status | provider |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1 |
e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9 | ACTIVE |
octavia |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip addr
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast
state UP group default qlen 1000
link/ether fa:16:3e:dc:9e:61 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.11/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fedc:9e61/64 scope link
validlft forever preferredlft forever

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.1 dev eth1 onlink
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.11
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.11
local 10.0.1.9 dev eth1 table local proto kernel scope host src
10.0.1.11
local 10.0.1.11 dev eth1 table local proto kernel scope host src
10.0.1.11
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.11
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:fedc:9e61 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
0: from all lookup local
100: from 10.0.1.9 lookup 1
32766: from all lookup main
32767: from all lookup default

If there is no router, the packets are supposed to forwarded in layer 2,
as amphora is plugged with ports of every member's subnet. The priority of
“from 10.0.1.9 lookup 1” is higher than that of "from all lookup local".
Maybe this patch (https://review.openstack.org/#/c/501915/) affects the
l2 traffic.

Best regards,
Yipei

On Sat, Nov 11, 2017 at 11:24 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I tried to run to command, and I think the amphora can connect to the
member (10.0.1.3). The results are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ping 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmpseq=1 ttl=64 time=189 ms
64 bytes from 10.0.1.3: icmp
seq=2 ttl=64 time=1.72 ms
^C
--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.722/95.855/189.989/94.134 ms

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy curl 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
Welcome to 10.0.1.3

stack@devstack-1:~$ sudo ip netns exec qdhcp-310fea4b-36ae-4617-b499-5936e8eda842
curl 10.0.1.3
Welcome to 10.0.1.3

As mentioned in my previous mail, I also have an environment installed
with octavia alone, where the error reproduces. In that environment, I also
tried the above commands, and have the same results. The member can be
reached from host and amphora.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy curl 10.0.1.5
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
Welcome to 10.0.1.5

stack@stack-VirtualBox:~$ sudo ip netns exec
qdhcp-13185eec-0996-4a08-b353-6775d5926b4c curl 10.0.1.5
Welcome to 10.0.1.5

In this environment, haproxy also tries to find the gateway ip 10.0.1.1.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C08:58:53.997948 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330593 ecr
0,nop,wscale 7], length 0
08:58:54.011771 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:54.991600 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:55.018542 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330850 ecr
0,nop,wscale 7], length 0
08:58:55.991703 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.015187 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.034507 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910331354 ecr
0,nop,wscale 7], length 0
08:58:58.016438 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.017650 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.115960 ARP, Request who-has 10.0.1.9 tell 10.0.1.2, length 28
08:58:59.116293 ARP, Reply 10.0.1.9 is-at fa:16:3e:dc:9e:61, length 28
08:59:01.031434 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:01.162845 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910332386 ecr
0,nop,wscale 7], length 0
08:59:02.031095 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:03.035527 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

15 packets captured
15 packets received by filter
0 packets dropped by kernel

And the gateway ip requested by haproxy is the same as it in the subnet.
+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

Some other info in this octavia env is as follows.

The info of load balancer:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| adminstateup | True |
| description | |
| id | d087d3b4-afbe-4af6-8b31-5e86fc97da1b |
| listeners | {"id": "d22644d2-cc40-44f1-b37f-2bb4f555f9b9"} |
| name | lb1 |
| operatingstatus | ONLINE |
| pools | {"id": "bc95c8e0-8475-4d97-9606-76c431e78ef7"} |
| provider | octavia |
| provisioning
status | ACTIVE |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| vip
address | 10.0.1.9 |
| vipportid | 902a78e7-a618-455b-91c7-cd36595475cc |
| vipsubnetid | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
+---------------------+------------------------------------------------+

The info of VMs:
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| 50ae3581-94fc-43ce-b53c-728715cd5597 | amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
| ACTIVE | - | Running | lb-mgmt-net=192.168.0.10;
net1=10.0.1.11 |
| d19b565d-14aa-4679-9d98-ff51461cd625 | vm1
| ACTIVE | - | Running | net1=10.0.1.5
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+

The info of listener:
+---------------------------+-------------------------------
-----------------+
| Field | Value
|
+---------------------------+-------------------------------
-----------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | bc95c8e0-8475-4d97-9606-76c431e78ef7
|
| default
tlscontainerref |
|
| description |
|
| id | d22644d2-cc40-44f1-b37f-2bb4f555f9b9
|
| loadbalancers | {"id": "d087d3b4-afbe-4af6-8b31-5e86fc97da1b"}
|
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | e59bb8f3bf9342aba02f9ba5804ed2fb
|
+---------------------------+-------------------------------
-----------------+

So I think the amphora can reach the member, it just cannot respond to
curl, hence failing to balance load across members. Maybe there is
something wrong with the image of the amphora. Since I replaced the latest
amphora image with an old one (built on 2017-07-24), it works. Totally
same environment except the amphora image.

Best regards,
Yipei

On Fri, Nov 10, 2017 at 5:24 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr
0,nop,wscale 7], length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr
0,nop,wscale 7], length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

*So if the subnet is not attached to any router, why does haproxy try to
find the gateway ip that does not exist at all? Maybe that is the reason
why haproxy receives the packet from curl but fail to respond. *

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle (https://wiki.openstack.org/wi
ki/Tricircle) are installed together. Because of the cross-neutron
mechanism of tricircle, the gateway ip of subnet in that region is
10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or 10.0.1.10) does
not exist in the network, since there is no router at all. This error also
happens in my another environment where octavia is installed alone. The
environment is installed on Oct. 6, and all the repos are the latest at
that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit
    message is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 | amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
    | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4;
    net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+-------------------------------------
    -----------+
    | Field | Value |
    +---------------------+-------------------------------------
    -----------+
    | adminstateup | True |
    | description | |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334
    |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"}
    |
    | name | lb1 |
    | operatingstatus | ONLINE |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"}
    |
    | provider | octavia |
    | provisioning
    status | ACTIVE |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | vip
    address | 10.0.1.4 |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b
    |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf
    |
    +---------------------+-------------------------------------
    -----------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1 |
    cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start":
    "10.0.1.1", "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as
    follows, respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------
    ------------+
    | Field | Value
    |
    +-------------------+---------------------------------------
    ------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"}
    |
    | | {"start": "192.168.1.10", "end": "192.168.1.254"}
    |
    | cidr | 192.168.1.0/24
    |
    | created
    at | 2017-11-05T12:14:45Z
    |
    | description |
    |
    | dnsnameservers |
    |
    | enable
    dhcp | True
    |
    | gatewayip | 192.168.1.9
    |
    | host
    routes |
    |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1
    |
    | ipversion | 4
    |
    | ipv6
    addressmode |
    |
    | ipv6
    ramode |
    |
    | name | lb-mgmt-subnet1
    |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea
    |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | revision
    number | 0
    |
    | servicetypes |
    |
    | subnetpool
    id |
    |
    | tags |
    |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | updated
    at | 2017-11-05T12:14:45Z
    |
    +-------------------+---------------------------------------
    ------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally
    return a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window
    irtt Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0
    0 o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0
    0 enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0
    0 enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0
    0 enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0
    0 o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0
    0 enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0
    0 virbr0

  3. In the amphora, the first two commit message of amphora-agent are
    as follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Yipei_Niu (2,060 points)   4
0 votes

Hi, Michael,

Please ignore my last two mails. Sorry about that.

The results of the two commands are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.10 dev eth1
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.8
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.8
local 10.0.1.4 dev eth1 table local proto kernel scope host src
10.0.1.8
local 10.0.1.8 dev eth1 table local proto kernel scope host src
10.0.1.8
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.8
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:febe:5ad5 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
0: from all lookup local
100: from 10.0.1.4 lookup 1
32766: from all lookup main
32767: from all lookup default

I think I know the source. When haproxy receives packets sent by curl, it
responds with taking VIP as source ip for 3-way handshake. Before adding “
100: from 10.0.1.9 lookup 1”, the datagrams are routed based on main table.
After adding "100: from 10.0.1.9 lookup 1", the haproxy tries to find the
gateway based on "default via 10.0.1.1 dev eth1 table 1 onlink". However,
if there is no router, the gateway ip is missing, making haproxy fails to
build tcp connection.

Best regards,
Yipei

On Tue, Nov 14, 2017 at 11:04 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Sorry about the typo in the last mail. Please just ignore the last mail.

In the environment where octavia and tricircle are installed together, I
created a router and attached subnet1 to it. Then I bind the mac address of
10.0.1.10 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
manually making amphora knows the mac address of 10.0.1.1 (actually it is
the mac of 10.0.1.10, since 10.0.1.1 does not exist), it works.

I also run the commands in this environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.10 dev eth1
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.8
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.8
local 10.0.1.4 dev eth1 table local proto kernel scope host src
10.0.1.8
local 10.0.1.8 dev eth1 table local proto kernel scope host src
10.0.1.8
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.8
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:febe:5ad5 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
0: from all lookup local
100: from 10.0.1.4 lookup 1
32766: from all lookup main
32767: from all lookup default

To make the situation clear, I run the commands in the environment
installed octavia alone. Please note that in this environment, there is no
router. The results are as follows.

stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| id | name | tenantid
| vip
address | provisioning_status | provider |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1 |
e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9 | ACTIVE |
octavia |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip addr
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast
state UP group default qlen 1000
link/ether fa:16:3e:dc:9e:61 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.11/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fedc:9e61/64 scope link
validlft forever preferredlft forever

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.1 dev eth1 onlink
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.11
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.11
local 10.0.1.9 dev eth1 table local proto kernel scope host src
10.0.1.11
local 10.0.1.11 dev eth1 table local proto kernel scope host src
10.0.1.11
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.11
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:fedc:9e61 dev lo table local proto none metric 0
pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
0: from all lookup local
100: from 10.0.1.9 lookup 1
32766: from all lookup main
32767: from all lookup default

If there is no router, the packets are supposed to forwarded in layer 2,
as amphora is plugged with ports of every member's subnet. The priority of
“from 10.0.1.9 lookup 1” is higher than that of "from all lookup local".
Maybe this patch (https://review.openstack.org/#/c/501915/) affects the
l2 traffic.

Best regards,
Yipei

On Tue, Nov 14, 2017 at 10:40 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your comments.

In the environment where octavia and tricircle are installed together, I
created a router and attached subnet1 to it. Then I bind the mac address of
10.0.1.9 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
manually making amphora knows the mac address of 10.0.1.1 (actually it is
the mac of 10.0.1.9, since 10.0.1.1 does not exist), it works.

To make the situation clear, I run the commands in the environment
installed octavia alone. Please note that in this environment, there is no
router. The results are as follows.

stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| id | name | tenantid
| vip
address | provisioning_status | provider |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+
| d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1 |
e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9 | ACTIVE |
octavia |
+--------------------------------------+------+-------------
---------------------+-------------+---------------------+----------+

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy ip addr
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1450 qdisc pfifofast
state UP group default qlen 1000
link/ether fa:16:3e:dc:9e:61 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.11/24 brd 10.0.1.255 scope global eth1
validlft forever preferredlft forever
inet 10.0.1.9/24 brd 10.0.1.255 scope global secondary eth1:0
validlft forever preferredlft forever
inet6 fe80::f816:3eff:fedc:9e61/64 scope link
validlft forever preferredlft forever

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
default via 10.0.1.1 dev eth1 table 1 onlink
default via 10.0.1.1 dev eth1 onlink
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.11
broadcast 10.0.1.0 dev eth1 table local proto kernel scope link src
10.0.1.11
local 10.0.1.9 dev eth1 table local proto kernel scope host src
10.0.1.11
local 10.0.1.11 dev eth1 table local proto kernel scope host src
10.0.1.11
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src
10.0.1.11
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium
local fe80::f816:3eff:fedc:9e61 dev lo table local proto none metric
0 pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
unreachable default dev lo table unspec proto kernel metric 4294967295
<0429%20496%207295> error -101 pref medium

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy ip rule show
sudo: unable to resolve host amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
0: from all lookup local
100: from 10.0.1.9 lookup 1
32766: from all lookup main
32767: from all lookup default

If there is no router, the packets are supposed to forwarded in layer 2,
as amphora is plugged with ports of every member's subnet. The priority of
“from 10.0.1.9 lookup 1” is higher than that of "from all lookup local".
Maybe this patch (https://review.openstack.org/#/c/501915/) affects the
l2 traffic.

Best regards,
Yipei

On Sat, Nov 11, 2017 at 11:24 AM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

I tried to run to command, and I think the amphora can connect to the
member (10.0.1.3). The results are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ping 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmpseq=1 ttl=64 time=189 ms
64 bytes from 10.0.1.3: icmp
seq=2 ttl=64 time=1.72 ms
^C
--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.722/95.855/189.989/94.134 ms

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy curl 10.0.1.3
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
Welcome to 10.0.1.3

stack@devstack-1:~$ sudo ip netns exec qdhcp-310fea4b-36ae-4617-b499-5936e8eda842
curl 10.0.1.3
Welcome to 10.0.1.3

As mentioned in my previous mail, I also have an environment installed
with octavia alone, where the error reproduces. In that environment, I also
tried the above commands, and have the same results. The member can be
reached from host and amphora.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy curl 10.0.1.5
sudo: unable to resolve host amphora-dcbff58a-d418-4271-937
4-9de2fd063ce9
Welcome to 10.0.1.5

stack@stack-VirtualBox:~$ sudo ip netns exec
qdhcp-13185eec-0996-4a08-b353-6775d5926b4c curl 10.0.1.5
Welcome to 10.0.1.5

In this environment, haproxy also tries to find the gateway ip 10.0.1.1.

ubuntu@amphora-dcbff58a-d418-4271-9374-9de2fd063ce9:~$ sudo ip netns
exec amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-dcbff58a-d418-4271-937
4-9de2fd063ce9
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C08:58:53.997948 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330593 ecr
0,nop,wscale 7], length 0
08:58:54.011771 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:54.991600 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:55.018542 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910330850 ecr
0,nop,wscale 7], length 0
08:58:55.991703 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.015187 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:57.034507 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910331354 ecr
0,nop,wscale 7], length 0
08:58:58.016438 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.017650 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:58:59.115960 ARP, Request who-has 10.0.1.9 tell 10.0.1.2, length 28
08:58:59.116293 ARP, Reply 10.0.1.9 is-at fa:16:3e:dc:9e:61, length 28
08:59:01.031434 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:01.162845 IP 10.0.1.2.55110 > 10.0.1.9.80: Flags [S], seq
4080898035, win 28200, options [mss 1410,sackOK,TS val 3910332386 ecr
0,nop,wscale 7], length 0
08:59:02.031095 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
08:59:03.035527 ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

15 packets captured
15 packets received by filter
0 packets dropped by kernel

And the gateway ip requested by haproxy is the same as it in the subnet.
+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocationpools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-10-08T06:33:09Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.1 |
| host
routes | |
| id | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 13185eec-0996-4a08-b353-6775d5926b4c |
| projectid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| updated
at | 2017-10-08T06:33:09Z |
+-------------------+--------------------------------------------+

Some other info in this octavia env is as follows.

The info of load balancer:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| adminstateup | True |
| description | |
| id | d087d3b4-afbe-4af6-8b31-5e86fc97da1b |
| listeners | {"id": "d22644d2-cc40-44f1-b37f-2bb4f555f9b9"} |
| name | lb1 |
| operatingstatus | ONLINE |
| pools | {"id": "bc95c8e0-8475-4d97-9606-76c431e78ef7"} |
| provider | octavia |
| provisioning
status | ACTIVE |
| tenantid | e59bb8f3bf9342aba02f9ba5804ed2fb |
| vip
address | 10.0.1.9 |
| vipportid | 902a78e7-a618-455b-91c7-cd36595475cc |
| vipsubnetid | 37023e56-a8bf-4070-8022-f6b6bb7b8e82 |
+---------------------+------------------------------------------------+

The info of VMs:
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| ID | Name
| Status | Task State | Power State | Networks
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+
| 50ae3581-94fc-43ce-b53c-728715cd5597 | amphora-dcbff58a-d418-4271-9374-9de2fd063ce9
| ACTIVE | - | Running | lb-mgmt-net=192.168.0.10;
net1=10.0.1.11 |
| d19b565d-14aa-4679-9d98-ff51461cd625 | vm1
| ACTIVE | - | Running | net1=10.0.1.5
|
+--------------------------------------+--------------------
--------------------------+--------+------------+-----------
--+------------------------------------------+

The info of listener:
+---------------------------+-------------------------------
-----------------+
| Field | Value
|
+---------------------------+-------------------------------
-----------------+
| adminstateup | True
|
| connectionlimit | -1
|
| default
poolid | bc95c8e0-8475-4d97-9606-76c431e78ef7
|
| default
tlscontainerref |
|
| description |
|
| id | d22644d2-cc40-44f1-b37f-2bb4f555f9b9
|
| loadbalancers | {"id": "d087d3b4-afbe-4af6-8b31-5e86fc97da1b"}
|
| name | listener1
|
| protocol | HTTP
|
| protocolport | 80
|
| sni
containerrefs |
|
| tenant
id | e59bb8f3bf9342aba02f9ba5804ed2fb
|
+---------------------------+-------------------------------
-----------------+

So I think the amphora can reach the member, it just cannot respond to
curl, hence failing to balance load across members. Maybe there is
something wrong with the image of the amphora. Since I replaced the latest
amphora image with an old one (built on 2017-07-24), it works. Totally
same environment except the amphora image.

Best regards,
Yipei

On Fri, Nov 10, 2017 at 5:24 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Thanks a lot for your reply.

I can make sure that there is no router or multiple dhcp services in my
environment.

As shown in my first mail, the haproxy in the amphora tries to find the
gateway ip 10.0.1.1 that does not exist in the environment.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy tcpdump -i eth1 -nn
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144
bytes
^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr
0,nop,wscale 7], length 0
07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr
0,nop,wscale 7], length 0
07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq
1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr
0,nop,wscale 7], length 0
07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28
07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28
07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28
07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28

*So if the subnet is not attached to any router, why does haproxy try
to find the gateway ip that does not exist at all? Maybe that is the reason
why haproxy receives the packet from curl but fail to respond. *

I think the gateway ip (10.0.1.10) confuses you. Actually, in my
environment octavia and tricircle (https://wiki.openstack.org/wi
ki/Tricircle) are installed together. Because of the cross-neutron
mechanism of tricircle, the gateway ip of subnet in that region is
10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or 10.0.1.10) does
not exist in the network, since there is no router at all. This error also
happens in my another environment where octavia is installed alone. The
environment is installed on Oct. 6, and all the repos are the latest at
that time.

Best regards,
Yipei

On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu newypei@gmail.com wrote:

Hi, Michael,

Based on your mail, the information is as follows.

  1. The version of Octavia I used is Queens, and the latest commit
    message is
    commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
    Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
    Date: Fri Nov 3 17:58:59 2017 +0000

    Updated from global requirements

    Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

  2. The info of the amphora and other VMs is as follows.
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | ID | Name
    | Status | Task State | Power State | Networks
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+
    | 33bd02cb-f853-404d-a705-99bc1b04a178 |
    amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f | ACTIVE | - |
    Running | lb-mgmt-net1=192.168.1.4; net1=10.0.1.8 |
    | dd046fc9-e2bf-437d-8c51-c397bccc3dc1 | client1
    | ACTIVE | - | Running | net1=10.0.1.3
    |
    | 50446c75-7cb7-43eb-b057-4b6b89a926bc | client3
    | ACTIVE | - | Running | net4=10.0.4.3
    |
    +--------------------------------------+--------------------
    --------------------------+--------+------------+-----------
    --+---------------------------------------------------------+

  3. The info of the load balancer is as follows.
    +---------------------+-------------------------------------
    -----------+
    | Field | Value
    |
    +---------------------+-------------------------------------
    -----------+
    | adminstateup | True
    |
    | description |
    |
    | id | 51cba1d5-cc3c-48ff-b41e-839619093334
    |
    | listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e57ecd20"}
    |
    | name | lb1
    |
    | operatingstatus | ONLINE
    |
    | pools | {"id": "d0042605-da50-4048-b298-660420b0a1d2"}
    |
    | provider | octavia
    |
    | provisioning
    status | ACTIVE
    |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | vip
    address | 10.0.1.4
    |
    | vipportid | 2209a819-0ac8-4211-b878-f0b41ac4727b
    |
    | vipsubnetid | cbcf4f04-da6d-4800-8b40-4b141972c2bf
    |
    +---------------------+-------------------------------------
    -----------+

  4. The info of the listener is as follows.
    +---------------------------+-------------------------------
    -----------------+
    | Field | Value
    |
    +---------------------------+-------------------------------
    -----------------+
    | adminstateup | True
    |
    | connectionlimit | -1
    |
    | default
    poolid | d0042605-da50-4048-b298-660420b0a1d2
    |
    | default
    tlscontainerref |
    |
    | description |
    |
    | id | b20ad920-c6cd-4e71-a9b9-c134e57ecd20
    |
    | loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-839619093334"}
    |
    | name | listener1
    |
    | protocol | HTTP
    |
    | protocolport | 80
    |
    | sni
    containerrefs |
    |
    | tenant
    id | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    +---------------------------+-------------------------------
    -----------------+

  5. The members of the load balancer lb1 are as follows.
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | id | name | tenantid
    | address | protocol
    port | weight | subnetid
    | admin
    state_up |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+
    | 420c905c-1077-46c9-8b04-526a59d93376 | |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.3 | 80 | 1
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | True |
    +--------------------------------------+------+-------------
    ---------------------+----------+---------------+--------+--
    ------------------------------------+----------------+

  6. Since the VIP and the members reside in the same subnet, only two
    subnets are listed as follows.
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | id | name | tenantid
    | cidr | allocation
    pools
    |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+
    | 752f865d-89e4-4284-9e91-8617a5a21da1 | lb-mgmt-subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 192.168.1.0/24 | {"start":
    "192.168.1.10", "end": "192.168.1.254"} |
    | | |
    | | {"start": "192.168.1.1", "end":
    "192.168.1.8"} |
    | cbcf4f04-da6d-4800-8b40-4b141972c2bf | subnet1 |
    c2a97a04cb6d4f25bdcb8b3f263c869e | 10.0.1.0/24 | {"start":
    "10.0.1.1", "end": "10.0.1.9"} |
    | | |
    | | {"start": "10.0.1.11", "end":
    "10.0.1.254"} |
    +--------------------------------------+-----------------+--
    --------------------------------+----------------+----------
    -----------------------------------------+

  7. The detailed info of subnet1 and lb-mgmt-subnet is listed as
    follows, respectively.
    lb-mgmt-subnet1
    +-------------------+---------------------------------------
    ------------+
    | Field | Value
    |
    +-------------------+---------------------------------------
    ------------+
    | allocationpools | {"start": "192.168.1.1", "end": "192.168.1.8"}
    |
    | | {"start": "192.168.1.10", "end":
    "192.168.1.254"} |
    | cidr | 192.168.1.0/24
    |
    | created
    at | 2017-11-05T12:14:45Z
    |
    | description |
    |
    | dnsnameservers |
    |
    | enable
    dhcp | True
    |
    | gatewayip | 192.168.1.9
    |
    | host
    routes |
    |
    | id | 752f865d-89e4-4284-9e91-8617a5a21da1
    |
    | ipversion | 4
    |
    | ipv6
    addressmode |
    |
    | ipv6
    ramode |
    |
    | name | lb-mgmt-subnet1
    |
    | network
    id | b4261144-3342-4605-8ca6-146e5b84c4ea
    |
    | projectid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | revision
    number | 0
    |
    | servicetypes |
    |
    | subnetpool
    id |
    |
    | tags |
    |
    | tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e
    |
    | updated
    at | 2017-11-05T12:14:45Z
    |
    +-------------------+---------------------------------------
    ------------+

subnet1
+-------------------+---------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------+
| allocationpools | {"start": "10.0.1.1", "end": "10.0.1.9"} |
| | {"start": "10.0.1.11", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| created
at | 2017-11-05T12:37:56Z |
| description | |
| dnsnameservers | |
| enable
dhcp | True |
| gatewayip | 10.0.1.10 |
| host
routes | |
| id | cbcf4f04-da6d-4800-8b40-4b141972c2bf |
| ipversion | 4 |
| ipv6
addressmode | |
| ipv6
ramode | |
| name | subnet1 |
| network
id | 310fea4b-36ae-4617-b499-5936e8eda842 |
| projectid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| revision
number | 0 |
| servicetypes | |
| subnetpool
id | |
| tags | |
| tenantid | c2a97a04cb6d4f25bdcb8b3f263c869e |
| updated
at | 2017-11-05T12:37:56Z |
+-------------------+---------------------------------------------+

  1. The info of interfaces in the default and amphora-haproxy network
    namespace of the amphora are as follows.
    ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ ifconfig
    ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77
    inet addr:192.168.1.4 Bcast:192.168.1.255
    Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
    RX packets:13112 errors:0 dropped:0 overruns:0 frame:0
    TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:128 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns
exec amphora-haproxy ifconfig
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e
e-05b695e2b71f
eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:218 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)

eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

  1. When curl the VIP from the host, it does not respond and finally
    return a timeout error.
    stack@devstack-1:/opt/stack/octavia$ sudo ip netns exec
    qdhcp-310fea4b-36ae-4617-b499-5936e8eda842 curl 10.0.1.4
    curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out

  2. Results of running "netstat -rn" on the host are as follows.
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window
    irtt Iface
    0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0
    0 o-hm0
    0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0
    0 enp0s3
    10.0.2.0 0.0.0.0 255.255.255.0 U 0 0
    0 enp0s3
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0
    0 enp0s10
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0
    0 o-hm0
    192.168.56.0 0.0.0.0 255.255.255.0 U 0 0
    0 enp0s10
    192.168.122.0 0.0.0.0 255.255.255.0 U 0 0
    0 virbr0

  3. In the amphora, the first two commit message of amphora-agent are
    as follows.

commit 2ab2836d0ebdd0fd5bc32d3adcc44a92557c8c1d
Author: OpenStack Proposal Bot openstack-infra@lists.openstack.org
Date: Fri Nov 3 17:58:59 2017 +0000

Updated from global requirements

Change-Id: I9047e289b8a3c931156da480b3f9f676c54a8358

commit 504cb6c682e4779b5889c0eb68705d0ab12e2c81
Merge: e983508 b8ebbe9
Author: Zuul zuul@review.openstack.org
Date: Wed Nov 1 19:46:39 2017 +0000

Merge "Add cached_zone to the amphora record"

Best regards,
Yipei


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Yipei_Niu (2,060 points)   4
...