settingsLogin | Registersettings

[Openstack-operators] Live migration failures

0 votes

After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live migration to work when a volume is attached. As it turns out when a live migration takes place the libvirt driver rewrites portions of the xml definition for the destination hypervisor and gets it wrong. Here is an example.

It turns this:

disk device="disk" type="network">










de400476-b68a-45a2-b04f-739313f42bef


Into this:












de400476-b68a-45a2-b04f-739313f42bef


Note the address change, libvirt will attempt to put this undefined disk at 0:0:0 which conflicts with the root disk and causes the migration to fail. While this may be a libvirt bug I can’t imagine anything good would come from changing the scsi address of your disks mid flight so steps should be taken in nova if rewriting the definition to preserve the address.

Here is my question, when exactly would you want to change the definition of an attached volume?


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Nov 2, 2017 in openstack-operators by Mike_Lowe (1,060 points)   3 3

4 Responses

0 votes

Hi Mike,

Which version of OpenStack are you using ? Do the normal migrations work ?

I found a bug that looks like your problem [1]

[1] https://bugs.launchpad.net/nova/+bug/1715569

Cheers,
Sergio

On 2 November 2017 at 08:48, Mike Lowe jomlowe@iu.edu wrote:

After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live
migration to work when a volume is attached. As it turns out when a live
migration takes place the libvirt driver rewrites portions of the xml
definition for the destination hypervisor and gets it wrong. Here is an
example.

It turns this:

disk device="disk" type="network">










de400476-b68a-45a2-b04f-739313f42bef


Into this:












de400476-b68a-45a2-b04f-739313f42bef


Note the address change, libvirt will attempt to put this undefined disk
at 0:0:0 which conflicts with the root disk and causes the migration to
fail. While this may be a libvirt bug I can’t imagine anything good would
come from changing the scsi address of your disks mid flight so steps
should be taken in nova if rewriting the definition to preserve the address.

Here is my question, when exactly would you want to change the definition
of an attached volume?


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
* Sergio Cuéllar │DevOps Engineer*
KIO NETWORKS
Mexico City Phone (52) 55 8503 2600 ext. 4335
Mobile: 5544844298
www.kionetworks.com

--


*“Este mensaje es confidencial. Si usted no es el destinatario de este
mensaje, le suplicamos se lo notifique al remitente mediante un correo
electrónico y que borre el presente mensaje y sus anexos de su computadora
sin retener una copia de los mismos. No debe copiar este mensaje o usarlo
para cualquier propósito ni divulgar su contenido. KIO Networks se reserva
el derecho de monitorear todas las comunicaciones de correo electrónico
(relacionadas o no con KIO Networks) que se transmitan a través de su
sistema. **Muchas gracias.*

This email is confidential and may also be privileged. If you are not the
intended recipient please immediately advise the sender by reply e-mail and
delete this message and its attachments from your computer without
retaining a copy. You should not copy it or use it for any purpose nor
disclose its contents to any other person. KIO Networks reserves the right
to monitor all email communications (whether related to the business of KIO
Networks or not) through its networks. Thank you.”



OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 2, 2017 by Sergio_Cuellar_Valde (240 points)  
0 votes

Look at comment #2, it’s me.

On Nov 2, 2017, at 11:34 AM, Sergio Cuellar Valdes scuellarv@kionetworks.com wrote:

Hi Mike,

Which version of OpenStack are you using ? Do the normal migrations work ?

I found a bug that looks like your problem [1]

[1] https://bugs.launchpad.net/nova/+bug/1715569 https://bugs.launchpad.net/nova/+bug/1715569

Cheers,
Sergio

On 2 November 2017 at 08:48, Mike Lowe <jomlowe@iu.edu jomlowe@iu.edu> wrote:
After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live migration to work when a volume is attached. As it turns out when a live migration takes place the libvirt driver rewrites portions of the xml definition for the destination hypervisor and gets it wrong. Here is an example.

It turns this:

disk device="disk" type="network">










de400476-b68a-45a2-b04f-739313f42bef


Into this:












de400476-b68a-45a2-b04f-739313f42bef


Note the address change, libvirt will attempt to put this undefined disk at 0:0:0 which conflicts with the root disk and causes the migration to fail. While this may be a libvirt bug I can’t imagine anything good would come from changing the scsi address of your disks mid flight so steps should be taken in nova if rewriting the definition to preserve the address.

Here is my question, when exactly would you want to change the definition of an attached volume?


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Sergio Cuéllar │DevOps Engineer
KIO NETWORKS
Mexico City Phone (52) 55 8503 2600 ext. 4335
Mobile: 5544844298
www.kionetworks.com


“Este mensaje es confidencial. Si usted no es el destinatario de este mensaje, le suplicamos se lo notifique al remitente mediante un correo electrónico y que borre el presente mensaje y sus anexos de su computadora sin retener una copia de los mismos. No debe copiar este mensaje o usarlo para cualquier propósito ni divulgar su contenido. KIO Networks se reserva el derecho de monitorear todas las comunicaciones de correo electrónico (relacionadas o no con KIO Networks) que se transmitan a través de su sistema. Muchas gracias.
This email is confidential and may also be privileged. If you are not the intended recipient please immediately advise the sender by reply e-mail and delete this message and its attachments from your computer without retaining a copy. You should not copy it or use it for any purpose nor disclose its contents to any other person. KIO Networks reserves the right to monitor all email communications (whether related to the business of KIO Networks or not) through its networks. Thank you.”

***********************************************************************************************************************_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 2, 2017 by Mike_Lowe (1,060 points)   3 3
0 votes

On 11/02/2017 08:48 AM, Mike Lowe wrote:
After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live migration to work when a volume is attached. As it turns out when a live migration takes place the libvirt driver rewrites portions of the xml definition for the destination hypervisor and gets it wrong. Here is an example.

Did you change versions of OpenStack as well?

Chris


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 2, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

I did not, it appears to be related to changing libvirt versions. It’s no longer as tolerant of being sloppy with device addresses. The last change for this section of code was a year ago if I read the git blame correctly.

On Nov 2, 2017, at 2:11 PM, Chris Friesen chris.friesen@windriver.com wrote:

On 11/02/2017 08:48 AM, Mike Lowe wrote:

After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live migration to work when a volume is attached. As it turns out when a live migration takes place the libvirt driver rewrites portions of the xml definition for the destination hypervisor and gets it wrong. Here is an example.

Did you change versions of OpenStack as well?

Chris


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 2, 2017 by Mike_Lowe (1,060 points)   3 3
...