settingsLogin | Registersettings

[openstack-dev] [nova] [ocata] nova-api error 404 instance not found

0 votes

Hi,

i've upgraded from newton to ocata and i my last problem (i hope so) is
in the nova-api.
if i try to create a new instance (horizon or cli) i get an "404
instance no found" error but is instance is created and started as
well.

nova-api.log:

2017-10-10 08:16:31.682 6 DEBUG nova.compute.api [req-4d354485-cfe9-
40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] [instance:
e564c631-896c-458c-93ab-b1c88f444fff] Fetching instance by UUID get
/usr/lib/python2.7/dist-packages/nova/compute/api.py:2360
2017-10-10 08:16:31.785 6 INFO nova.api.openstack.wsgi [req-4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] HTTP exception
thrown: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be
found.
2017-10-10 08:16:31.787 6 DEBUG nova.api.openstack.wsgi [req-4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] Returning 404 to
user: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be found.
call /usr/lib/python2.7/dist-
packages/nova/api/openstack/wsgi.py:1039

Does anybody know whats the problem?

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Oct 11, 2017 in openstack-dev by Kim-Norman_Sahm (320 points)   4 5

4 Responses

0 votes

On 10/10/2017 3:53 AM, Kim-Norman Sahm wrote:
Hi,

i've upgraded from newton to ocata and i my last problem (i hope so) is
in the nova-api.
if i try to create a new instance (horizon or cli) i get an "404
instance no found" error but is instance is created and started as
well.

nova-api.log:

2017-10-10 08:16:31.682 6 DEBUG nova.compute.api [req-4d354485-cfe9-
40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] [instance:
e564c631-896c-458c-93ab-b1c88f444fff] Fetching instance by UUID get
/usr/lib/python2.7/dist-packages/nova/compute/api.py:2360
2017-10-10 08:16:31.785 6 INFO nova.api.openstack.wsgi [req-4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] HTTP exception
thrown: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be
found.
2017-10-10 08:16:31.787 6 DEBUG nova.api.openstack.wsgi [req-4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] Returning 404 to
user: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be found.
call /usr/lib/python2.7/dist-
packages/nova/api/openstack/wsgi.py:1039

Does anybody know whats the problem?

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Do you have all of the latest fixes for Ocata? You should be at version
15.0.7 if you're on the latest Ocata fix release.

Is the instance properly mapped to a cell? Try querying the nova_api
database:

select * from novaapi.instancemappings where instance_uuid =
"e564c631-896c-458c-93ab-b1c88f444fff";

Did the instance go to ACTIVE state or did it fail to schedule? If it
failed to schedule, it would be in the nova_cell0 database and should be
in ERROR state. You could query the cell0 database with:

select * from nova_cell0.instances where uuid =
"e564c631-896c-458c-93ab-b1c88f444fff";

Otherwise see #3 in the Cells FAQs page here:

https://docs.openstack.org/nova/latest/user/cells#faqs

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 10, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 9
0 votes

Hi Matt,

i'm using the ubuntu packages v15.0.6
the instances are mapped to the correct cell:

mysql> select * from novaapi.instancemappings where instanceuuid =
-> "e564c631-896c-458c-93ab-b1c88f444fff";
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+
| created
at | updatedat | id |
instance
uuid | cellid |
project
id |
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+
| 2017-10-10 08:16:32 | 2017-10-10 08:16:32 | 846 | e564c631-896c-458c-
93ab-b1c88f444fff | 6 | 7c1dd7d33037481e81f55d2f5d45bb90 |
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+

I've tried the workaround of this bug:
https://bugs.launchpad.net/nova/+bug/1682423

and its running correctly.

br
Kim

Am Dienstag, den 10.10.2017, 09:08 -0500 schrieb Matt Riedemann:

On 10/10/2017 3:53 AM, Kim-Norman Sahm wrote:

Hi,

i've upgraded from newton to ocata and i my last problem (i hope
so) is
in the nova-api.
if i try to create a new instance (horizon or cli) i get an "404
instance no found" error but is instance is created and started as
well.

nova-api.log:

2017-10-10 08:16:31.682 6 DEBUG nova.compute.api [req-4d354485-
cfe9-
40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] [instance:
e564c631-896c-458c-93ab-b1c88f444fff] Fetching instance by UUID get
/usr/lib/python2.7/dist-packages/nova/compute/api.py:2360
2017-10-10 08:16:31.785 6 INFO nova.api.openstack.wsgi [req-
4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] HTTP exception
thrown: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be
found.
2017-10-10 08:16:31.787 6 DEBUG nova.api.openstack.wsgi [req-
4d354485-
cfe9-40f2-8507-a48e02db0af0 db88dca6f68c4ab2b82dbad7476bb122
7c1dd7d33037481e81f55d2f5d45bb90 - default default] Returning 404
to
user: Instance e564c631-896c-458c-93ab-b1c88f444fff could not be
found.
call /usr/lib/python2.7/dist-
packages/nova/api/openstack/wsgi.py:1039

Does anybody know whats the problem?

Best regards
Kim

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB
17689



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Do you have all of the latest fixes for Ocata? You should be at
version
15.0.7 if you're on the latest Ocata fix release.

Is the instance properly mapped to a cell? Try querying the nova_api
database:

select * from novaapi.instancemappings where instance_uuid =
"e564c631-896c-458c-93ab-b1c88f444fff";

Did the instance go to ACTIVE state or did it fail to schedule? If
it
failed to schedule, it would be in the nova_cell0 database and should
be
in ERROR state. You could query the cell0 database with:

select * from nova_cell0.instances where uuid =
"e564c631-896c-458c-93ab-b1c88f444fff";

Otherwise see #3 in the Cells FAQs page here:

https://docs.openstack.org/nova/latest/user/cells#faqs

Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.sahm@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Oct 11, 2017 by Kim-Norman_Sahm (320 points)   4 5
0 votes

On 10/11/2017 3:48 AM, Kim-Norman Sahm wrote:
Hi Matt,

i'm using the ubuntu packages v15.0.6
the instances are mapped to the correct cell:

mysql> select * from novaapi.instancemappings where instanceuuid =
-> "e564c631-896c-458c-93ab-b1c88f444fff";
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+
| created
at | updatedat | id |
instance
uuid | cellid |
project
id |
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+
| 2017-10-10 08:16:32 | 2017-10-10 08:16:32 | 846 | e564c631-896c-458c-
93ab-b1c88f444fff | 6 | 7c1dd7d33037481e81f55d2f5d45bb90 |
+---------------------+---------------------+-----+------------------
--------------------+---------+----------------------------------+

I've tried the workaround of this bug:
https://bugs.launchpad.net/nova/+bug/1682423

and its running correctly.

br
Kim

I put some comments in the bug report (starting at comment 16). I don't
really see how this would be happening unless there is an issue with the
osapi_compute service version, like if you're running the API with uwsgi
(which wasn't supported in ocata) or if you're API services are not all
upgraded yet.

The workaround with the 1 second sleep seems to suggest there is a race
window where the instance is not mapped to a cell yet but the build
request is also gone, which shouldn't happen as we create the instance
in the cell before deleting the build request:

https://github.com/openstack/nova/blob/15.0.0/nova/conductor/manager.py#L914

https://github.com/openstack/nova/blob/15.0.0/nova/conductor/manager.py#L937

So while it's possible for the instance to be created in the cell
database but the instance mapping record isn't yet updated, we account
for that when looking up the instance here:

https://github.com/openstack/nova/blob/15.0.0/nova/compute/api.py#L2276

and here:

https://github.com/openstack/nova/blob/15.0.0/nova/compute/api.py#L2289

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 11, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 9
0 votes

On 10/12/2017 6:56 AM, Kim-Norman Sahm wrote:
You're right.
i've used the "old" wsgi based nova api.
after switch to nova-api binary is looks fine.

Thanks
Kim

Great. I think we should probably delete those old unsupported wsgi
files in stable/ocata since they won't work. When they were originally
added we didn't have code that depended on the osapi_compute service
version but we do in Ocata so using those is actually just broken
deployment waiting to happen.

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 12, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 9
...