settingsLogin | Registersettings

[openstack-dev] Future-based api for openstack clients calls, that starts background tasks

0 votes

Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many third-party
scripts.

I propose to provide common higth-level API for such functions, which uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async(....)
.....
vm = create_future.result()

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
--
Kostiantyn Danilov aka koder http://koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150112/4012e18b/attachment.html

asked Jan 12, 2015 in openstack-dev by Konstantin_Danilov (540 points)   2
retagged Jan 28, 2015 by admin

4 Responses

0 votes

Konstantin,

I believe it's better to work on server side, and use some modern approach
like web sockets for async operations. So we won't need to retrieve 100500
times information about object. And then use this feature in clients.

createfuture = novaclient.servers.createasync(....)
.....
vm = create_future.result()

As well this pattern doesn't look great.

I would prefer to see something like:

vm = novaclient.servers.create(...., sync=True)

Best regards,
Boris Pavlovic

On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov
wrote:

Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many third-party
scripts.

I propose to provide common higth-level API for such functions, which uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async(....)
.....
vm = create_future.result()

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
--
Kostiantyn Danilov aka koder
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jan 12, 2015 by Boris_Pavlovic (2,540 points)   2 5
0 votes

Boris,

Move from sync http to something like websocket requires a lot of work and
not directly connected
with API issue. When openstack api servers begin to support
websockets - it would be easy to change implementation of monitoring thread
without breaking compatibility.
At the moment periodical pooling from additional thread looks reasonable
for me
and it creates same amount of http requests as all current implementation.

BP is not about improving performance,
but about providing convenient and common API to handle background tasks.

So we won't need to retrieve 100500 times information about object.
As I sad before - this API creates same amount of load as any code which we
use to check background task currently.
It even can decrease load due to requests aggregation in some cases (but
there a points to discuss).

As well this pattern doesn't look great.
I would prefer to see something like:
vm = novaclient.servers.create(...., sync=True)

This is completely different pattern. It is blocking call, which don't
allows you to start two(or more) background tasks
and from same thread and make some calculations while they running in
background.

On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic
wrote:

Konstantin,

I believe it's better to work on server side, and use some modern approach
like web sockets for async operations. So we won't need to retrieve 100500
times information about object. And then use this feature in clients.

createfuture = novaclient.servers.createasync(....)

.....
vm = create_future.result()

As well this pattern doesn't look great.

I would prefer to see something like:

vm = novaclient.servers.create(...., sync=True)

Best regards,
Boris Pavlovic

On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov <kdanilov at mirantis.com

wrote:

Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many
third-party scripts.

I propose to provide common higth-level API for such functions, which uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async(....)
.....
vm = create_future.result()

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
--
Kostiantyn Danilov aka koder
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jan 12, 2015 by Konstantin_Danilov (540 points)   2
0 votes

On Mon, Jan 12, 2015 at 10:17 PM, Konstantin Danilov
wrote:

Boris,

Move from sync http to something like websocket requires a lot of work and
not directly connected
with API issue. When openstack api servers begin to support
websockets - it would be easy to change implementation of monitoring
thread without breaking compatibility.
At the moment periodical pooling from additional thread looks reasonable
for me
and it creates same amount of http requests as all current implementation.

BP is not about improving performance,
but about providing convenient and common API to handle background tasks.

So we won't need to retrieve 100500 times information about object.
As I sad before - this API creates same amount of load as any code which
we use to check background task currently.
It even can decrease load due to requests aggregation in some cases (but
there a points to discuss).

As well this pattern doesn't look great.
I would prefer to see something like:
vm = novaclient.servers.create(...., sync=True)

This is completely different pattern. It is blocking call, which don't
allows you to start two(or more) background tasks
and from same thread and make some calculations while they running in
background.

Except if you use threads (eventlet or other) - I am still struggling to
enjoy Futures/yield based flowcontrol, lost battle i guess:(.

On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic
wrote:

Konstantin,

I believe it's better to work on server side, and use some modern
approach like web sockets for async operations. So we won't need to
retrieve 100500 times information about object. And then use this feature
in clients.

createfuture = novaclient.servers.createasync(....)

.....
vm = create_future.result()

As well this pattern doesn't look great.

I would prefer to see something like:

vm = novaclient.servers.create(...., sync=True)

Best regards,
Boris Pavlovic

On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov <
kdanilov at mirantis.com> wrote:

Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those
functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many
third-party scripts.

We have some very similar code at the moment, but we are keen to move away
from it to
something like making use of rpc ".{start,end}" notifications to reduce the
load we put on keystone and
friends.

I propose to provide common higth-level API for such functions, which
uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way
to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async(....)
.....
vm = create_future.result()

Is that going to return on any state change or do you pass in a list of
acceptable states?

-Angus

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
--
Kostiantyn Danilov aka koder
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jan 12, 2015 by Angus_Salkeld (5,520 points)   1 4 5
0 votes

On Mon, Jan 12, 2015 at 3:33 PM, Angus Salkeld
wrote:

On Mon, Jan 12, 2015 at 10:17 PM, Konstantin Danilov <
kdanilov at mirantis.com> wrote:

Boris,

Move from sync http to something like websocket requires a lot of work
and not directly connected
with API issue. When openstack api servers begin to support
websockets - it would be easy to change implementation of monitoring
thread without breaking compatibility.
At the moment periodical pooling from additional thread looks reasonable
for me
and it creates same amount of http requests as all current
implementation.

BP is not about improving performance,
but about providing convenient and common API to handle background tasks.

So we won't need to retrieve 100500 times information about object.
As I sad before - this API creates same amount of load as any code which
we use to check background task currently.
It even can decrease load due to requests aggregation in some cases (but
there a points to discuss).

As well this pattern doesn't look great.
I would prefer to see something like:
vm = novaclient.servers.create(...., sync=True)

This is completely different pattern. It is blocking call, which don't
allows you to start two(or more) background tasks
and from same thread and make some calculations while they running in
background.

Except if you use threads (eventlet or other) - I am still struggling to
enjoy Futures/yield based flowcontrol, lost battle i guess:(.

On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic
wrote:

Konstantin,

I believe it's better to work on server side, and use some modern
approach like web sockets for async operations. So we won't need to
retrieve 100500 times information about object. And then use this feature
in clients.

createfuture = novaclient.servers.createasync(....)

.....
vm = create_future.result()

As well this pattern doesn't look great.

I would prefer to see something like:

vm = novaclient.servers.create(...., sync=True)

Best regards,
Boris Pavlovic

On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov <
kdanilov at mirantis.com> wrote:

Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those
functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many
third-party scripts.

We have some very similar code at the moment, but we are keen to move away
from it to
something like making use of rpc ".{start,end}" notifications to reduce
the load we put on keystone and
friends.

This is nice approach for core projects, yet novaclient users are typically
can't use such approach.

But the nice things about futures is that we can have different engines
(websockets, sync http, rpc with callbacks, etc)
behind same API, and hide all implementation details behind it. It's even
possible to use them in simultaneously -
different engine would used to handle different calls.

I propose to provide common higth-level API for such functions, which
uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way
to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async(....)
.....
vm = create_future.result()

Is that going to return on any state change or do you pass in a list of
acceptable states?

In general it should returns result if background task is completed
successfully or raise exception
if it fails. For server I currently use 'active' as success marker and
'error' or timeout for exception
( https://github.com/koder-ua/os_api/blob/master/os_api/nova.py#L74 ), and
hope that expected states
can be calculated from api call/parameters automatically, but not 100%
sure. So - yes, additional
parameter might be required.

-Angus

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
--
Kostiantyn Danilov aka koder
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jan 12, 2015 by Konstantin_Danilov (540 points)   2
...