settingsLogin | Registersettings

[openstack-dev] [openstack-ansible] Live migration issue

0 votes

Hi osa team,

i ‘ve got live migration issue in one direction but not in other.
I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag.

My 2 compute node are same host type and have nova-compute and cinder-volume (our ceph cluster as backend) services.

No problem to live migrate instance from Compute 2 to Compute 1 whereas the reverse is not true.
See log below:

Live migration instance Compute 2 to 1: OK

Compute 2 log
2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event)
2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed
2017-01-25 11:00:17.795 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] postlivemigration() is started..
2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmpfL96lI/privsep.sock']
2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon running as pid 28815
2017-01-25 11:00:18.397 28309 INFO nova.compute.manager [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync
powerstate the instance has a pending task (migrating). Skip.
2017-01-25 11:00:18.538 28309 INFO nova.compute.manager [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Running instance usage audit for host p-oscompute02 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 2 instances.
2017-01-25 11:00:18.691 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:19.634 28309 INFO nova.compute.resourcetracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:19.635 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:19.709 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:20.163 28309 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:20.201 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:00:20.202 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete
2017-01-25 11:00:20.352 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:21.143 28309 INFO nova.compute.resource
tracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:21.143 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:21.204 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:21.214 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migrating instance to p-oscompute01 finished successfully.
2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored.
2017-01-25 11:00:33.399 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Stopped (Lifecycle Event)

Compute 1 log
2017-01-25 11:00:13.067 113231 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:00:13.434 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmp0444cm/privsep.sock']
2017-01-25 11:00:13.984 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep daemon running as pid 113978
2017-01-25 11:00:14.856 113231 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:15.931 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Auditing locally available compute resources for node p-oscompute01.openstack.local
2017-01-25 11:00:16.748 113231 INFO nova.compute.resource
tracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Total usable vcpus: 40, total allocated vcpus: 8
2017-01-25 11:00:16.749 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Final resource view: name=p-oscompute01.openstack.local physram=128700MB usedram=12288MB physdisk=33493GB useddisk=72GB totalvcpus=40 usedvcpus=8 pcistats=[]
2017-01-25 11:00:16.819 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Computeservice record updated for p-oscompute01:p-oscompute01.openstack.local
2017-01-25 11:00:17.227 113231 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Started (Lifecycle Event)
2017-01-25 11:00:17.896 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:17.959 113231 INFO nova.compute.manager [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Running instance usage audit for host p-oscompute01 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 7 instances.
2017-01-25 11:00:18.027 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the syncpower process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:18.028 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:18.135 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the sync
power process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:20.231 113231 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Post operation of migration started

Live migration Compute 1 to 2: NOK

Compute 1 log
2017-01-25 11:03:57.676 113231 INFO nova.virt.libvirt.migration [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:03:57.812 113231 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:03:58.475 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Live Migration failure: Requested operation is not valid: domain 'instance-00000187' is already active
2017-01-25 11:03:58.817 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has aborted

Compute 2 log
2017-01-25 11:03:55.783 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:03:56.849 28309 INFO os
vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserve_on_delete=False,vif_name='tapdff20b91-a6') 2017-01-25 11:03:58.981 28309 INFO nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detach volume f40efa24-4992-49ac-8d75-4ace88d9ecf7 from mountpoint /dev/vda 2017-01-25 11:03:58.984 28309 WARNING nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detaching volume from unknown instance 2017-01-25 11:03:58.986 28309 WARNING nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During detach_volume, instance disappeared. 2017-01-25 11:04:00.033 28309 INFO nova.virt.libvirt.driver [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During wait destroy, instance disappeared. 2017-01-25 11:04:00.034 28309 INFO os_vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridge_name='brqc434ace8-45',has_traffic_filtering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linux_bridge',port_profile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:04:00.049 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:04:00.050 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete

Need some help or some hints to resolve/debug this issue. Give me some headache ;-) Because compute node hardware are the same, nova/libvirt conf deployed with osa.

Regards,

Fabrice Grelaud
Université de Bordeaux


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jan 25, 2017 in openstack-dev by Fabrice_Grelaud (780 points)   3 5

2 Responses

0 votes

Hi,

What domain name are you using?
Check for 'Traceback' and ' ERROR ' in the logs, maybe you will get a hint

2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored.

-----Original Message-----
From: fabrice grelaud [mailto:fabrice.grelaud@u-bordeaux.fr]
Sent: Wednesday, January 25, 2017 1:06 PM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: [openstack-dev] [openstack-ansible] Live migration issue

Hi osa team,

i ‘ve got live migration issue in one direction but not in other.
I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag.

My 2 compute node are same host type and have nova-compute and cinder-volume (our ceph cluster as backend) services.

No problem to live migrate instance from Compute 2 to Compute 1 whereas the reverse is not true.
See log below:

Live migration instance Compute 2 to 1: OK

Compute 2 log
2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event)
2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed
2017-01-25 11:00:17.795 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] postlivemigration() is started..
2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmpfL96lI/privsep.sock']
2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon running as pid 28815
2017-01-25 11:00:18.397 28309 INFO nova.compute.manager [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync
powerstate the instance has a pending task (migrating). Skip.
2017-01-25 11:00:18.538 28309 INFO nova.compute.manager [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Running instance usage audit for host p-oscompute02 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 2 instances.
2017-01-25 11:00:18.691 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:19.634 28309 INFO nova.compute.resourcetracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:19.635 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:19.709 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:20.163 28309 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:20.201 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:00:20.202 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete
2017-01-25 11:00:20.352 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:21.143 28309 INFO nova.compute.resource
tracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:21.143 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:21.204 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:21.214 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migrating instance to p-oscompute01 finished successfully.
2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored.
2017-01-25 11:00:33.399 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Stopped (Lifecycle Event)

Compute 1 log
2017-01-25 11:00:13.067 113231 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:00:13.434 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmp0444cm/privsep.sock']
2017-01-25 11:00:13.984 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep daemon running as pid 113978
2017-01-25 11:00:14.856 113231 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:15.931 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Auditing locally available compute resources for node p-oscompute01.openstack.local
2017-01-25 11:00:16.748 113231 INFO nova.compute.resource
tracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Total usable vcpus: 40, total allocated vcpus: 8
2017-01-25 11:00:16.749 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Final resource view: name=p-oscompute01.openstack.local physram=128700MB usedram=12288MB physdisk=33493GB useddisk=72GB totalvcpus=40 usedvcpus=8 pcistats=[]
2017-01-25 11:00:16.819 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Computeservice record updated for p-oscompute01:p-oscompute01.openstack.local
2017-01-25 11:00:17.227 113231 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Started (Lifecycle Event)
2017-01-25 11:00:17.896 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:17.959 113231 INFO nova.compute.manager [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Running instance usage audit for host p-oscompute01 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 7 instances.
2017-01-25 11:00:18.027 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the syncpower process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:18.028 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:18.135 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the sync
power process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:20.231 113231 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Post operation of migration started

Live migration Compute 1 to 2: NOK

Compute 1 log
2017-01-25 11:03:57.676 113231 INFO nova.virt.libvirt.migration [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:03:57.812 113231 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:03:58.475 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Live Migration failure: Requested operation is not valid: domain 'instance-00000187' is already active
2017-01-25 11:03:58.817 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has aborted

Compute 2 log
2017-01-25 11:03:55.783 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:03:56.849 28309 INFO os
vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserve_on_delete=False,vif_name='tapdff20b91-a6') 2017-01-25 11:03:58.981 28309 INFO nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detach volume f40efa24-4992-49ac-8d75-4ace88d9ecf7 from mountpoint /dev/vda 2017-01-25 11:03:58.984 28309 WARNING nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detaching volume from unknown instance 2017-01-25 11:03:58.986 28309 WARNING nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During detach_volume, instance disappeared. 2017-01-25 11:04:00.033 28309 INFO nova.virt.libvirt.driver [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During wait destroy, instance disappeared. 2017-01-25 11:04:00.034 28309 INFO os_vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridge_name='brqc434ace8-45',has_traffic_filtering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linux_bridge',port_profile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:04:00.049 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:04:00.050 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete

Need some help or some hints to resolve/debug this issue. Give me some headache ;-) Because compute node hardware are the same, nova/libvirt conf deployed with osa.

Regards,

Fabrice Grelaud
Université de Bordeaux


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 25, 2017 by Lenny_Verkhovsky (1,460 points)   3
0 votes

Thanks for reply.

But « a priori » log say « this error can be safely ignore ». And therefore, this log comes from the live migration that succeeded (compute 2 to compute 1).

The ERROR that questions me is (live migration compute 1 to 2), on compute 1:
2017-01-25 11:03:58.475 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Live Migration failure: Requested operation is not valid: domain 'instance-00000187' is already active

Le 25 janv. 2017 à 13:24, Lenny Verkhovsky lennyb@mellanox.com a écrit :

Hi,

What domain name are you using?
Check for 'Traceback' and ' ERROR ' in the logs, maybe you will get a hint

2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored.

-----Original Message-----
From: fabrice grelaud [mailto:fabrice.grelaud@u-bordeaux.fr]
Sent: Wednesday, January 25, 2017 1:06 PM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: [openstack-dev] [openstack-ansible] Live migration issue

Hi osa team,

i ‘ve got live migration issue in one direction but not in other.
I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag.

My 2 compute node are same host type and have nova-compute and cinder-volume (our ceph cluster as backend) services.

No problem to live migrate instance from Compute 2 to Compute 1 whereas the reverse is not true.
See log below:

Live migration instance Compute 2 to 1: OK

Compute 2 log
2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event)
2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed
2017-01-25 11:00:17.795 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] postlivemigration() is started..
2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmpfL96lI/privsep.sock']
2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon running as pid 28815
2017-01-25 11:00:18.397 28309 INFO nova.compute.manager [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync
powerstate the instance has a pending task (migrating). Skip.
2017-01-25 11:00:18.538 28309 INFO nova.compute.manager [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Running instance usage audit for host p-oscompute02 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 2 instances.
2017-01-25 11:00:18.691 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:19.634 28309 INFO nova.compute.resourcetracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:19.635 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:19.709 28309 INFO nova.compute.resource
tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:20.163 28309 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:20.201 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:00:20.202 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete
2017-01-25 11:00:20.352 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local
2017-01-25 11:00:21.143 28309 INFO nova.compute.resource
tracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Total usable vcpus: 40, total allocated vcpus: 4
2017-01-25 11:00:21.143 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Final resource view: name=p-oscompute02.openstack.local physram=128700MB usedram=6144MB physdisk=33493GB useddisk=22GB totalvcpus=40 usedvcpus=4 pcistats=[]
2017-01-25 11:00:21.204 28309 INFO nova.compute.resourcetracker [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Computeservice record updated for p-oscompute02:p-oscompute02.openstack.local
2017-01-25 11:00:21.214 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migrating instance to p-oscompute01 finished successfully.
2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored.
2017-01-25 11:00:33.399 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Stopped (Lifecycle Event)

Compute 1 log
2017-01-25 11:00:13.067 113231 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:00:13.434 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep
context', 'osbrick.privileged.default', '--privsepsockpath', '/tmp/tmp0444cm/privsep.sock']
2017-01-25 11:00:13.984 113231 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-01-25 11:00:13.986 113231 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP
SYSADMIN/CAPSYSADMIN/none
2017-01-25 11:00:13.987 113231 INFO oslo.privsep.daemon [-] privsep daemon running as pid 113978
2017-01-25 11:00:14.856 113231 INFO os
vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:00:15.931 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Auditing locally available compute resources for node p-oscompute01.openstack.local
2017-01-25 11:00:16.748 113231 INFO nova.compute.resource
tracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Total usable vcpus: 40, total allocated vcpus: 8
2017-01-25 11:00:16.749 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Final resource view: name=p-oscompute01.openstack.local physram=128700MB usedram=12288MB physdisk=33493GB useddisk=72GB totalvcpus=40 usedvcpus=8 pcistats=[]
2017-01-25 11:00:16.819 113231 INFO nova.compute.resourcetracker [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Computeservice record updated for p-oscompute01:p-oscompute01.openstack.local
2017-01-25 11:00:17.227 113231 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Started (Lifecycle Event)
2017-01-25 11:00:17.896 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:17.959 113231 INFO nova.compute.manager [req-7d438646-5f1d-4cad-9c29-98272c1b8352 - - - - -] Running instance usage audit for host p-oscompute01 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 7 instances.
2017-01-25 11:00:18.027 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the syncpower process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:18.028 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Resumed (Lifecycle Event)
2017-01-25 11:00:18.135 113231 INFO nova.compute.manager [req-926f2642-43af-42a6-8b0a-1fcbf8c433ee - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During the sync
power process the instance has moved from host p-oscompute02 to host p-oscompute01
2017-01-25 11:00:20.231 113231 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Post operation of migration started

Live migration Compute 1 to 2: NOK

Compute 1 log
2017-01-25 11:03:57.676 113231 INFO nova.virt.libvirt.migration [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time
2017-01-25 11:03:57.812 113231 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
2017-01-25 11:03:58.475 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Live Migration failure: Requested operation is not valid: domain 'instance-00000187' is already active
2017-01-25 11:03:58.817 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has aborted

Compute 2 log
2017-01-25 11:03:55.783 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Instance launched has CPU info: {"vendor": "Intel", "model": "Haswell-noTSX", "arch": "x8664", "features": ["pge", "avx", "clflush", "sep", "syscall", "vme", "dtes64", "invpcid", "tsc", "fsgsbase", "xsave", "vmx", "erms", "xtpr", "cmov", "smep", "ssse3", "est", "pat", "monitor", "smx", "pbe", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "pclmuldq", "acpi", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "tm2", "ht", "dca", "lahflm", "abm", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "pse", "ds", "invtsc", "pni", "rdtscp", "avx2", "aes", "sse2", "ss", "dscpl", "bmi1", "bmi2", "pcid", "fpu", "cx16", "pse36", "mtrr", "movbe", "pdcm", "rdrand", "x2apic"], "topology": {"cores": 10, "cells": 2, "threads": 2, "sockets": 1}}
2017-01-25 11:03:56.849 28309 INFO os
vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridgename='brqc434ace8-45',hastrafficfiltering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linuxbridge',portprofile=<?>,preserve_on_delete=False,vif_name='tapdff20b91-a6') 2017-01-25 11:03:58.981 28309 INFO nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detach volume f40efa24-4992-49ac-8d75-4ace88d9ecf7 from mountpoint /dev/vda 2017-01-25 11:03:58.984 28309 WARNING nova.compute.manager [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Detaching volume from unknown instance 2017-01-25 11:03:58.986 28309 WARNING nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During detach_volume, instance disappeared. 2017-01-25 11:04:00.033 28309 INFO nova.virt.libvirt.driver [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During wait destroy, instance disappeared. 2017-01-25 11:04:00.034 28309 INFO os_vif [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridge_name='brqc434ace8-45',has_traffic_filtering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linux_bridge',port_profile=<?>,preserveondelete=False,vifname='tapdff20b91-a6')
2017-01-25 11:04:00.049 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3del
2017-01-25 11:04:00.050 28309 INFO nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deletion of /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3
del complete

Need some help or some hints to resolve/debug this issue. Give me some headache ;-) Because compute node hardware are the same, nova/libvirt conf deployed with osa.

Regards,

Fabrice Grelaud
Université de Bordeaux


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 25, 2017 by Fabrice_Grelaud (780 points)   3 5
...