settingsLogin | Registersettings

[Openstack] Floating IP not being added in namespace anymore

0 votes

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying._______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

asked Sep 19, 2017 in openstack by Jean-Philippe_Methot (600 points)   1 2 4

12 Responses

0 votes

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan jp.methot@planethoster.info a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc._______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
0 votes

Do you have this fix
https://review.openstack.org/#/c/501317/

Ajay

From: JP Japan jp.methot@planethoster.info
Date: Monday, September 18, 2017 at 5:02 PM
To: "openstack@lists.openstack.org" openstack@lists.openstack.org
Subject: Re: [Openstack] Floating IP not being added in namespace anymore

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan jp.methot@planethoster.info a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Ajay_Kalambur_(akala (2,680 points)   7 14
0 votes

I saw something similar did you restart all the services after the upgrade? Just wonder. I saw some other issue when I upgraded from 7.3 to 7.4 where it gave me some vif error after all servers reboot the problem has been gone.

Let me know.

Il giorno 18 set 2017, alle ore 17:02, JP Japan jp.methot@planethoster.info ha scritto:

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan jp.methot@planethoster.info a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded Sep 19, 2017 by Remo_Mattei (10,500 points)   1 2 4
0 votes

Hi,

I do not have this fix. Seems it’s too recent for the latest RDO-Ocata. I will apply it, it should solve the iptables issue. I have a hunch it’s not the cause of the missing floatingip issue though, but I will try.

Thank you for your help,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 09:51, Ajay Kalambur (akalambu) akalambu@cisco.com a écrit :

Do you have this fix
https://review.openstack.org/#/c/501317/ https://review.openstack.org/#/c/501317/

Ajay

From: JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info>
Date: Monday, September 18, 2017 at 5:02 PM
To: "openstack@lists.openstack.org openstack@lists.openstack.org" <openstack@lists.openstack.org openstack@lists.openstack.org>
Subject: Re: [Openstack] Floating IP not being added in namespace anymore

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
0 votes

Hi,

Thank you for your reply. We did restart all neutron services, several times. We also restarted the servers but the issue is still there.

Best regards,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:01, Remo Mattei remo@italy1.com a écrit :

I saw something similar did you restart all the services after the upgrade? Just wonder. I saw some other issue when I upgraded from 7.3 to 7.4 where it gave me some vif error after all servers reboot the problem has been gone.

Let me know.

Il giorno 18 set 2017, alle ore 17:02, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> ha scritto:

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
0 votes

are you running RDO / Juju? What is the version?

Thanks

On 9/18/17 6:40 PM, Jean-Philippe Méthot wrote:
Hi,

Thank you for your reply. We did restart all neutron services, several
times. We also restarted the servers but the issue is still there.

Best regards,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:01, Remo Mattei <remo@italy1.com
remo@italy1.com> a écrit :

I saw something similar did you restart all the services after the
upgrade? Just wonder. I saw some other issue when I upgraded from 7.3
to 7.4 where it gave me some vif error after all servers reboot the
problem has been gone. 

Let me know. 

Il giorno 18 set 2017, alle ore 17:02, JP Japan
<jp.methot@planethoster.info jp.methot@planethoster.info> ha
scritto:

Sorry, I ended up sending the previous email a bit too quickly.
Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info
jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production
infrastructure: we updated to latest Ocata and we changed the
outgoing port on our network node to a lacp port. We made the change
by switching the port in br-ex in openvswitch to the new lacp-backed
port. Ever since these two things happened right after the other,
we’ve ran into two issues, one which has much worse consequences
than the other:

1.We can’t add floating ips to instances anymore. The interface says
the operation completed successfully, the database gets updated, but
the IP address doesn’t exist in the network namespace on the network
nodes. Strangely enough, the iptables rules in the NAT table do
exist. The port just doesn’t receive the new address. Adding the
floating ip address manually to the virtual interface with "ip netns
exec qrouter namespace id ip addr add ip address dev virtual
interface
" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts
informing us it was unable to add some rules in iptables because
there’s a lock on xtables, while as far as we know, the L3-agent
itself is the one holding the lock. Here’s the error: 

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager #
Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I
neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp
-m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager #
Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ;
Stdout: ; Stderr: Another app is currently holding the xtables lock.
Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager 

It’s not clear exactly how this is affecting the setup, as metadata
is still going through properly (most likely through the DHCP) but
it’s quite worrying.


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Remo_Mattei (10,500 points)   1 2 4
0 votes

I use RDO Ocata without any deployment tool
Neutron version is openstack-neutron-10.0.3-1.el7.noarch

It's from August 28th.

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 11:00, Remo Mattei remo@italy1.com a écrit :

are you running RDO / Juju? What is the version?

Thanks

On 9/18/17 6:40 PM, Jean-Philippe Méthot wrote:

Hi,

Thank you for your reply. We did restart all neutron services, several times. We also restarted the servers but the issue is still there.

Best regards,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:01, Remo Mattei <remo@italy1.com remo@italy1.com> a écrit :

I saw something similar did you restart all the services after the upgrade? Just wonder. I saw some other issue when I upgraded from 7.3 to 7.4 where it gave me some vif error after all servers reboot the problem has been gone.

Let me know.

Il giorno 18 set 2017, alle ore 17:02, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> ha scritto:

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
0 votes

Ouch no deployment tools? Nevertheless I will check the version I have on mine

Remo

Il giorno 18 set 2017, alle ore 19:43, Jean-Philippe Méthot jp.methot@planethoster.info ha scritto:

I use RDO Ocata without any deployment tool
Neutron version is openstack-neutron-10.0.3-1.el7.noarch

It's from August 28th.

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 11:00, Remo Mattei remo@italy1.com a écrit :

are you running RDO / Juju? What is the version?

Thanks

On 9/18/17 6:40 PM, Jean-Philippe Méthot wrote:
Hi,

Thank you for your reply. We did restart all neutron services, several times. We also restarted the servers but the issue is still there.

Best regards,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:01, Remo Mattei remo@italy1.com a écrit :

I saw something similar did you restart all the services after the upgrade? Just wonder. I saw some other issue when I upgraded from 7.3 to 7.4 where it gave me some vif error after all servers reboot the problem has been gone.

Let me know.

Il giorno 18 set 2017, alle ore 17:02, JP Japan jp.methot@planethoster.info ha scritto:

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan jp.methot@planethoster.info a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded Sep 19, 2017 by Remo_Mattei (10,500 points)   1 2 4
0 votes

It seems that adding this patch did not solve the issue. I’m still getting those iptable errors.

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:37, Jean-Philippe Méthot jp.methot@planethoster.info a écrit :

Hi,

I do not have this fix. Seems it’s too recent for the latest RDO-Ocata. I will apply it, it should solve the iptables issue. I have a hunch it’s not the cause of the missing floatingip issue though, but I will try.

Thank you for your help,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 09:51, Ajay Kalambur (akalambu) <akalambu@cisco.com akalambu@cisco.com> a écrit :

Do you have this fix
https://review.openstack.org/#/c/501317/ https://review.openstack.org/#/c/501317/

Ajay

From: JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info>
Date: Monday, September 18, 2017 at 5:02 PM
To: "openstack@lists.openstack.org openstack@lists.openstack.org" <openstack@lists.openstack.org openstack@lists.openstack.org>
Subject: Re: [Openstack] Floating IP not being added in namespace anymore

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
0 votes

We fixed our floating ip problem, or at least we believe we did so. We cleaned out the neutron lock files and since then, no floating ip issues.

However, we’re still getting the iptables error messages on l3-agent boot.

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 14:08, Remo Mattei remo@italy1.com a écrit :

Ouch no deployment tools? Nevertheless I will check the version I have on mine

Remo

Il giorno 18 set 2017, alle ore 19:43, Jean-Philippe Méthot <jp.methot@planethoster.info jp.methot@planethoster.info> ha scritto:

I use RDO Ocata without any deployment tool
Neutron version is openstack-neutron-10.0.3-1.el7.noarch

It's from August 28th.

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 11:00, Remo Mattei <remo@italy1.com remo@italy1.com> a écrit :

are you running RDO / Juju? What is the version?

Thanks

On 9/18/17 6:40 PM, Jean-Philippe Méthot wrote:

Hi,

Thank you for your reply. We did restart all neutron services, several times. We also restarted the servers but the issue is still there.

Best regards,

Jean-Philippe Méthot
Openstack system administrator
Administrateur système Openstack
PlanetHoster inc.

Le 19 sept. 2017 à 10:01, Remo Mattei <remo@italy1.com remo@italy1.com> a écrit :

I saw something similar did you restart all the services after the upgrade? Just wonder. I saw some other issue when I upgraded from 7.3 to 7.4 where it gave me some vif error after all servers reboot the problem has been gone.

Let me know.

Il giorno 18 set 2017, alle ore 17:02, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> ha scritto:

Sorry, I ended up sending the previous email a bit too quickly. Here’s some more info about our setup.

-It’s running latest Ocata with Openvswitch and network dedicated nodes.
-The network nodes are L3HA
-There’s no DVR here.

Le 19 sept. 2017 à 08:51, JP Japan <jp.methot@planethoster.info jp.methot@planethoster.info> a écrit :

Hi,

A few days ago, we made two big changes on our production infrastructure: we updated to latest Ocata and we changed the outgoing port on our network node to a lacp port. We made the change by switching the port in br-ex in openvswitch to the new lacp-backed port. Ever since these two things happened right after the other, we’ve ran into two issues, one which has much worse consequences than the other:

1.We can’t add floating ips to instances anymore. The interface says the operation completed successfully, the database gets updated, but the IP address doesn’t exist in the network namespace on the network nodes. Strangely enough, the iptables rules in the NAT table do exist. The port just doesn’t receive the new address. Adding the floating ip address manually to the virtual interface with "ip netns exec qrouter namespace id ip addr add ip address dev virtual interface" solves this, but is in no way a permanent solution.

2.We’re getting an error message in the L3-agent whenever it starts informing us it was unable to add some rules in iptables because there’s a lock on xtables, while as far as we know, the L3-agent itself is the one holding the lock. Here’s the error:

2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Generated by iptablesmanager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager *nat
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager -I neutron-l3-agent-PREROUTING 7 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager COMMIT
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager # Completed by iptables
manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager
2017-09-18 13:00:55.426 18575 ERROR neutron.callbacks.manager

It’s not clear exactly how this is affecting the setup, as metadata is still going through properly (most likely through the DHCP) but it’s quite worrying.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2017 by Jean-Philippe_Methot (600 points)   1 2 4
...