settingsLogin | Registersettings

[openstack-dev] [Neutron][LBaaS] LBaaSv2 with HAproxy Agent Deployment Issue

0 votes

All,

I am trying to add Neutron LBaaSv2 to a working OpenStack Liberty deployment. I am running into an issue where the lbaas agent does not appear in the output of neutron agent-list. However, the lbaas extension appears in the output of neutron ext-list. After investigating further, the lbaas-agent sends a message on the queue and times out waiting for a reply:

2016-06-06 21:09:15.958 22 INFO oslo.messaging.drivers.implrabbit [-] Connected to AMQP server on 10.32.20.52:5672
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager [-] Unable to retrieve ready devices
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager Traceback (most recent call last):
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/neutro
nlbaas/services/loadbalancer/agent/agentmanager.py", line 152, in syncstate
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager readyinstances = set(self.pluginrpc.getr
eadydevices())
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/neutro
n
lbaas/services/loadbalancer/agent/agentapi.py", line 36, in getreadydevices
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager return cctxt.call(self.context, 'getready_
devices', host=self.host)
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/rpc/client.py", line 158, in call
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager retry=self.retry)
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/transport.py", line 90, in send
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager timeout=timeout, retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/drivers/amqpdriver.py", line 431, in send
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager retry=retry)
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/drivers/amqpdriver.py", line 420, in _send
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager result = self.waiter.wait(msgid, timeout)
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/drivers/amqpdriver.py", line 318, in wait
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agentmanager message = self.waiters.get(msgid, timeout=
timeout)
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager File "/usr/lib/python2.7/site-packages/oslom
essaging/
drivers/amqpdriver.py", line 223, in get
2016-06-06 21:10:15.972 22 ERROR neutronlbaas.services.loadbalancer.agent.agentmanager 'to message ID %s' % msgid)
2016-06-06 21:10:15.972 22 ERROR neutron
lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply
to message ID eae3cc1bc8614aa8ae499d92ca4ec731

I verified that the lbaas queues reside within the Rabbit cluster:

bash-4.2$ rabbitmqctl list_queues

n-lbaasagent 0
n-lbaas
agent.control-server-1.novalocal 0
n-lbaasagent.control-server-2.novalocal 0
n-lbaas
agent.control-server-3.novalocal 0
n-lbaasagentfanout18a3b28c969148f3a008df8f3e5f5363 0
n-lbaas
agentfanouta7d48e8a1b27443d82ee4944bec44cf8 0
n-lbaasagentfanoutb5360edb19c240e79c71d60806977f66 0
n-lbaasv2-plugin 0
n-lbaasv2-plugin.control-server-1.novalocal 0
n-lbaasv2-plugin.control-server-2.novalocal 0
n-lbaasv2-plugin.control-server-3.novalocal 0
n-lbaasv2-plugin
fanout5cbb6dd4fafc4c4784add8a20e0a28a5 0
n-lbaasv2-plugin
fanout756ee4e4eee547528d0f6e3dde71b150 0
n-lbaasv2-plugin
fanout_7629f7bb85ce493d83c334dfcc2cd4aa 0
notifications.info 8

And the lbaas queues are being mirrored:

rabbitmq server logs

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25481.1>

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25635.1>

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:23 ===
Mirrored queue 'n-lbaasv2-pluginfanout659b460849ef43ee834ce6d88d294b46' in vhost '/': Synchronising: all slaves already synced
...
=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25503.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25657.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-3': <3038.25506.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25660.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanouta6c881c1ae9c4a259b6a3f8c2f42a0b4' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Adding mirror on node 'rabbit@mercury-control-se
rver-2': <3037.25660.1>

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: 0 messages to synchronise

=INFO REPORT==== 6-Jun-2016::19:01:24 ===
Mirrored queue 'n-lbaasv2-pluginfanout3d7abfb48a7a4bc8bbfb406490e9e8b6' in vhost '/': Synchronising: all slaves already synced

All other neutron agents and openstack services are functional and communicate across the same Rabbit cluster. [1] provides additional details of the setup. It appears I may be running into bug [2]. I added myself to this bug, but it seems to have gone dormant. I would appreciate any support in resolving this issue.

[1] https://gist.github.com/danehans/8691464051cb43feacac11afd4bbc447
[2] https://bugs.launchpad.net/neutron/+bug/1464241

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 8, 2016 in openstack-dev by Daneyon_Hansen_(dane (1,500 points)   2 7
retagged Jan 25, 2017 by admin
...