settingsLogin | Registersettings

[openstack-dev] [octavia] how to recreate amphora instances

0 votes

Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18 | ACTIVE | octavia |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7 | ACTIVE | octavia |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17 | ACTIVE | octavia |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24 | ACTIVE | octavia |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5 | ACTIVE | octavia |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23 | ACTIVE | octavia |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Nov 10, 2017 in openstack-dev by Kim-Norman_Sahm (320 points)   4 5

4 Responses

0 votes

I think we helped you get going again in the IRC channel. Please ping
us again in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm
kim-norman.sahm@noris.de wrote:
Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18 | ACTIVE | octavia |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7 | ACTIVE | octavia |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17 | ACTIVE | octavia |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24 | ACTIVE | octavia |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5 | ACTIVE | octavia |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23 | ACTIVE | octavia |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 6, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

I am also interested how to fix this. If you can describe shortly the procedure.

Thanks,
Mihaela

-----Original Message-----
From: Michael Johnson [mailto:johnsomor@gmail.com]
Sent: Monday, November 06, 2017 6:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I think we helped you get going again in the IRC channel. Please ping us again in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm kim-norman.sahm@noris.de wrote:
Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18 | ACTIVE | octavia |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7 | ACTIVE | octavia |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17 | ACTIVE | octavia |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24 | ACTIVE | octavia |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5 | ACTIVE | octavia |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23 | ACTIVE | octavia |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des
Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by mihaela.balas_at_ora (480 points)   1 2
0 votes

I can give it a go, there are also logs of our conversation on
evesdrop here: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking
was out. This caused the load balancer amphora to be detected as
failed and started the failover process which rebuilds the amphora,
however this process also failed due to the wider infrastructure
issues (Note, this is what I remember from the conversation. Correct
the background if I am wrong). At this point the load balancers would
be in a provisioning status of "ERROR", the amphora instances were
likely deleted (depending on where the infrastructure issue impacted
the failover process), and the amphora db records would be in the
"DELETED" state. To make their situation worse, the DB cleanup cycle
of the housekeeping process was set low (default is a week) and the
amphora records were purged. This meant the customer data for the VIP
address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure
issues are resolved, you can simply go into the octavia database and
issue: update loadbalancer set provisioningstatus="ACTIVE" where
provisioning_status = "ERROR";
This will restart the health manager expecting the amphora to report
health heartbeats and the failover process with start again rebuilding
the amphora. We have also recently added an admin API to trigger a
failover manually
(https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't
do this). This means that the VIP address and neutron port
information for that VIP address are not available for the failover
process to rebuild. In this case, before you start the above process
you will need to recreate the amphora records from your logs, either
adding the port information if the ports are still live in neutron, or
creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM, mihaela.balas@orange.com wrote:
I am also interested how to fix this. If you can describe shortly the procedure.

Thanks,
Mihaela

-----Original Message-----
From: Michael Johnson [mailto:johnsomor@gmail.com]
Sent: Monday, November 06, 2017 6:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I think we helped you get going again in the IRC channel. Please ping us again in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm kim-norman.sahm@noris.de wrote:

Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18 | ACTIVE | octavia |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7 | ACTIVE | octavia |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17 | ACTIVE | octavia |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24 | ACTIVE | octavia |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5 | ACTIVE | octavia |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23 | ACTIVE | octavia |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des
Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

Hi Michael,

Thank you for the detailed explanation. I was in the worst scenario where the database entries were purged and I had to manually re-create the DB entries and the ports. I successfully managed to insert the lines in the database and the amphoras were created.

Thanks a lot for the hints. I also increased the DB cleanup interval to 1 week.

Mihaela

-----Original Message-----
From: Michael Johnson [mailto:johnsomor@gmail.com]
Sent: Friday, November 10, 2017 3:31 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I can give it a go, there are also logs of our conversation on evesdrop here: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking was out. This caused the load balancer amphora to be detected as failed and started the failover process which rebuilds the amphora, however this process also failed due to the wider infrastructure issues (Note, this is what I remember from the conversation. Correct the background if I am wrong). At this point the load balancers would be in a provisioning status of "ERROR", the amphora instances were likely deleted (depending on where the infrastructure issue impacted the failover process), and the amphora db records would be in the "DELETED" state. To make their situation worse, the DB cleanup cycle of the housekeeping process was set low (default is a week) and the amphora records were purged. This meant the customer data for the VIP address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure issues are resolved, you can simply go into the octavia database and
issue: update loadbalancer set provisioningstatus="ACTIVE" where provisioning_status = "ERROR"; This will restart the health manager expecting the amphora to report health heartbeats and the failover process with start again rebuilding the amphora. We have also recently added an admin API to trigger a failover manually (https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't do this). This means that the VIP address and neutron port information for that VIP address are not available for the failover process to rebuild. In this case, before you start the above process you will need to recreate the amphora records from your logs, either adding the port information if the ports are still live in neutron, or creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM, mihaela.balas@orange.com wrote:
I am also interested how to fix this. If you can describe shortly the procedure.

Thanks,
Mihaela

-----Original Message-----
From: Michael Johnson [mailto:johnsomor@gmail.com]
Sent: Monday, November 06, 2017 6:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora
instances

I think we helped you get going again in the IRC channel. Please ping us again in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm kim-norman.sahm@noris.de wrote:

Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18 | ACTIVE | octavia |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7 | ACTIVE | octavia |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17 | ACTIVE | octavia |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24 | ACTIVE | octavia |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5 | ACTIVE | octavia |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3 | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23 | ACTIVE | octavia |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des
Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


_ ____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
exploites ou copies sans autorisation. Si vous avez recu ce message
par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or
privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 13, 2017 by mihaela.balas_at_ora (480 points)   1 2
...