settingsLogin | Registersettings

[openstack-dev] [nova] api.fault notification is never emitted

0 votes

Hi,

I come across a questionable behavior of nova while I tried to use the
notifyonapifaults configuration option [0] while testing the related
versioned notification transformation patch [1]. Based on the
description of the config option and the code that uses it [2] nova
sends and api.fault notification if the nova-api service encounters an
unhandle exception. There is a FaultWrapper class [3] added to the
pipeline of the REST request which catches every exception and calls
the notification sending.
Based on some debugging in devstack this FaultWrapper never catches any
exception. I injected a ValueError to the beginning of
nova.objects.aggregate.Aggregate.create method. This resulted in an
HTTPInternalServerError exception and HTTP 500 error code but the
exception handling part of the FaultWrapper [4] was never reached. So I
dig a bit deeper and I think I found the reason. Every REST API method
is decorated with expected
errors decorator [5] which as a last resort
translate the unexpected exception to HTTPInternalServerError. In the
wsgi stack the actual REST api call is guarded with
ResourceExceptionHandler context manager [7] which translates
HTTPException to a Fault [8]. Then Fault is catched and translated to
the REST response [7]. This way the exception never propagates back to
the FaultWrapper in [6] and therefore the api.fault notification is
never emitted.

You can see the api logs here [9] and the patch that I used to add the
extra traces here [10]. Please note that there is a compute.exception
notification visible in the log but that is a different notification
emitted from wrap_exception decorator [11] used in compute.manager [12]
and compute.api [13] only.

So my questions are:
1) Is it a bug in the nova wsgi or it is expected that the wsgi code
catches everything?
2) Do we need FaultWrapper at all if the wsgi stack catches every
exception?
3) Do we need api.fault notification at all? It seems nobody missed it
so far.
4) If we want to have api.fault notification then what would be the
good place to emit it? Maybe ResourceExceptionHandler at [8]?

I filed a bug for tracking purposes [14].

Cheers,
gibi

[0]
https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49
[1] https://review.openstack.org/#/c/469038
[2]
https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83
[3]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79
[4]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87
[5]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325
[6]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368
[7]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637
[8]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418
[9] https://pastebin.com/Eu6rBjNN
[10] https://pastebin.com/en4aFutc
[11]
https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57
[12]
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105
[13]
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92
[14] https://bugs.launchpad.net/nova/+bug/1699115


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 26, 2017 in openstack-dev by =?iso-8859-1?Q?Bal=E (5,020 points)   1 2 4

1 Response

0 votes

Cross-posting to the operators list since they are the ones that would
care about this.

Basically, the "notifyonapi_faults" config option hasn't worked since
probably Kilo when the 2.1 microversion wsgi stack code was added.

Rackspace added it back in 2012:

https://review.openstack.org/#/c/13288/

Gibi has a patch proposed to remove it since it's dead code:

https://review.openstack.org/#/c/505164/

Given how long this has been regressed, and no one has noticed, it seems
fair to just remove this.

Is anyone relying on this and for some reason disagrees with the bug and
that we should try to fix this?

On 6/20/2017 7:22 AM, Balazs Gibizer wrote:
Hi,

I come across a questionable behavior of nova while I tried to use the
notifyonapifaults configuration option [0] while testing the related
versioned notification transformation patch [1]. Based on the
description of the config option and the code that uses it [2] nova
sends and api.fault notification if the nova-api service encounters an
unhandle exception. There is a FaultWrapper class [3] added to the
pipeline of the REST request which catches every exception and calls the
notification sending.
Based on some debugging in devstack this FaultWrapper never catches any
exception. I injected a ValueError to the beginning of
nova.objects.aggregate.Aggregate.create method. This resulted in an
HTTPInternalServerError exception and HTTP 500 error code but the
exception handling part of the FaultWrapper [4] was never reached. So I
dig a bit deeper and I think I found the reason. Every REST API method
is decorated with expected
errors decorator [5] which as a last resort
translate the unexpected exception to HTTPInternalServerError. In the
wsgi stack the actual REST api call is guarded with
ResourceExceptionHandler context manager [7] which translates
HTTPException to a Fault [8]. Then Fault is catched and translated to
the REST response [7]. This way the exception never propagates back to
the FaultWrapper in [6] and therefore the api.fault notification is
never emitted.

You can see the api logs here [9] and the patch that I used to add the
extra traces here [10]. Please note that there is a compute.exception
notification visible in the log but that is a different notification
emitted from wrap_exception decorator [11] used in compute.manager [12]
and compute.api [13] only.

So my questions are:
1) Is it a bug in the nova wsgi or it is expected that the wsgi code
catches everything?
2) Do we need FaultWrapper at all if the wsgi stack catches every
exception?
3) Do we need api.fault notification at all? It seems nobody missed it
so far.
4) If we want to have api.fault notification then what would be the good
place to emit it? Maybe ResourceExceptionHandler at [8]?

I filed a bug for tracking purposes [14].

Cheers,
gibi

[0]
https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49

[1] https://review.openstack.org/#/c/469038
[2]
https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83

[3]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79

[4]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87

[5]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325

[6]
https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368

[7]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637

[8]
https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418

[9] https://pastebin.com/Eu6rBjNN
[10] https://pastebin.com/en4aFutc
[11]
https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57
[12]
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105
[13] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92
[14] https://bugs.launchpad.net/nova/+bug/1699115


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 26, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 10
...