settingsLogin | Registersettings

[Openstack] [nova]How to convince the RetryFilter that a host is recovered from a failure?

0 votes

Hi, forks:

I have a openstack o version cluster with two compute nodes. The resources
are almost used up.
So, I add a new compute node to a running cluster and try to cold migrate a
vm to the new compute node. At the first time, the migration failed with
'No valid host found' for my mistake. I forgot to mount the data disk to
extend local available storage.

After I extend the storage, I confirmed the hypervisor have recognized the
free disk space, as shown in the output of 'openstack hypervisor show'. But
when I try to cold migrate again, It failed with 'No valid host found'
again. After some dig in the scheduler log, I find the RetryFilter reject
the new added host.

My question are:
1. Where does the RetryFilter store the host availability status? In memory
or in DB? Which table?
2. How to convince the RetryFilter that the host is available again? I
tried to restart nova-api, nova-scheduler, memcached services, but it
doesn't work.

The related log lines as following:

2017-11-15 14:24:37.906 7 DEBUG oslomessaging.drivers.amqpdriver [-]
received message msgid: fa38b97af9504ebaa1d3118d91157f45 reply to
reply
2c3ce56f29774ed19c504407625c632c call
/var/lib/kolla/venv/lib/python2.7/site-packages/oslomessaging/drivers/amqpdriver.py:194
2017-11-15 14:24:37.999 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Starting with 0 host(s)
getfilteredobjects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
2017-11-15 14:24:38.000 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filter RetryFilter returned 0 hosts
2017-11-15 14:24:38.001 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filtering removed all hosts for the
request with instance ID '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter
results: [('RetryFilter', None)] getfilteredobjects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
2017-11-15 14:24:38.001 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filtering removed all hosts for the
request with instance ID '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter
results: ['RetryFilter: (start: 0, end: 0)']
2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filterscheduler
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] There are 0 hosts available but 1
instances requested to build. select
destinations
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76

Any help are appreciated.
Thanks

Sam Song


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Nov 15, 2017 in openstack by sam_song (180 points)   1 2

1 Response

0 votes

On Wed, Nov 15, 2017 at 9:59 AM, Song Sam samsong8610@gmail.com wrote:
Hi, forks:

I have a openstack o version cluster with two compute nodes. The
resources are almost used up.
So, I add a new compute node to a running cluster and try to cold
migrate a vm to the new compute node. At the first time, the
migration failed with 'No valid host found' for my mistake. I forgot
to mount the data disk to extend local available storage.

After I extend the storage, I confirmed the hypervisor have
recognized the free disk space, as shown in the output of 'openstack
hypervisor show'. But when I try to cold migrate again, It failed
with 'No valid host found' again. After some dig in the scheduler
log, I find the RetryFilter reject the new added host.

It seems to me that you hit this bug:
https://bugs.launchpad.net/nova/+bug/1718512

My question are:
1. Where does the RetryFilter store the host availability status? In
memory or in DB? Which table?

The RequestSpec persists the already tried hosts and RetryFilter checks
that.

  1. How to convince the RetryFilter that the host is available again?
    I tried to restart nova-api, nova-scheduler, memcached services, but
    it doesn't work.

There is a requestspec table in the novaapi db where the RequestSpec
objects are stored. Unfortunately the content of the object is json
serialized into the spec field of that table so I don't suggest to try
to fix it manually.

cheers,
gibi

The related log lines as following:

2017-11-15 14:24:37.906 7 DEBUG oslomessaging.drivers.amqpdriver
[-] received message msgid: fa38b97af9504ebaa1d3118d91157f45 reply
to reply
2c3ce56f29774ed19c504407625c632c call
/var/lib/kolla/venv/lib/python2.7/site-packages/oslomessaging/drivers/amqpdriver.py:194
2017-11-15 14:24:37.999 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Starting with 0 host(s) getfilteredobjects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
2017-11-15 14:24:38.000 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filter RetryFilter returned 0 hosts
2017-11-15 14:24:38.001 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
[('RetryFilter', None)] getfilteredobjects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
2017-11-15 14:24:38.001 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
['RetryFilter: (start: 0, end: 0)']
2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filterscheduler
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] There are 0 hosts available but 1 instances requested to build.
select
destinations
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76

Any help are appreciated.
Thanks

Sam Song


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 15, 2017 by =?iso-8859-1?Q?Bal=E (5,020 points)   1 2 3
...