settingsLogin | Registersettings

[openstack-dev] [python-openstacksdk] Status of python-openstacksdk project

0 votes

Hi OpenStack developers,

I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks,
Michael (johnsom)

[1]
https://review.openstack.org/#/q/project:openstack/python-openstacksdk+statu
s:open+topic:%255Eoctavia.*


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 7, 2017 in openstack-dev by Michael_Johnson (4,380 points)   4 5

12 Responses

0 votes

Michael Johnson wrote:
I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks for raising this.

Beyond its limited activity, another issue is that it's not an official
project while its name make it a "default choice" for a lot of users
(hard to blame them for thinking that
http://git.openstack.org/cgit/openstack/python-openstacksdk is not the
official Python SDK for OpenStack, but I digress). So I agree that the
situation should be clarified.

I know that Monty has pretty strong feelings about this too, so I'll
wait for him to comment.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Thierry_Carrez (57,480 points)   3 8 12
0 votes

On 08/04/2017 03:24 AM, Thierry Carrez wrote:
Michael Johnson wrote:

I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks for raising this.

Beyond its limited activity, another issue is that it's not an official
project while its name make it a "default choice" for a lot of users
(hard to blame them for thinking that
http://git.openstack.org/cgit/openstack/python-openstacksdk is not the
official Python SDK for OpenStack, but I digress). So I agree that the
situation should be clarified.

I know that Monty has pretty strong feelings about this too, so I'll
wait for him to comment.

Oh boy. I'd kind of hoped we'd make it to the PTG before starting this
conversation ... guess not. :)

Concerns


I share the same concerns Thierry listed above. Specifically:

  • It is not an official project, but its name leads people to believe
    it's the "right" thing to use if they want to talk to OpenStack clouds
    using Python.

  • The core team is small to begin with, but recently got hit in a major
    way by shifts in company priorities.

I think we can all agree that those are concerns.

Three additional points:

  • The OpenStack AppDev group and the various appdev hackathons use
    shade, not openstacksdk. This is what we have people out in the world
    recommending people use when they write code that consumes OpenStack
    APIs. The Interop challenges at the Summits so far have all used
    Ansible's OpenStack modules, which are based on shade, because they were
    the thing that works.

  • Both shade and python-openstackclient have investigated using
    openstacksdk as their REST layer but were unable to because it puts some
    abstractions in layers that make it impossible to do some of the
    advanced things we need.

  • openstacksdk has internal implementations of things that exist at
    different points in the stack. We just added full support for version
    service and version discovery to keystoneauth, but openstacksdk has its
    own layer for that so it both can't use the ksa implementation and is
    not compliant with the API-WG consume guidelines.

It's not all bad! There is some great work in openstacksdk and it's
a shame there are some things that make it hard to consume. Brian,
Qiming and Terry have done a bunch of excellent work - and I'd like to
not lose it to the dustbin of corporate shifting interest.

warning - there is a very large text wall that follows. If you don't
care a ton on this topic, please stop reading now, otherwise you might
rage-quit computers altogether.

Proposal


I'd propose we have the shade team adopt the python-openstacksdk codebase.

This is obviously an aggressive suggestion and essentially represents a
takeover of a project. We don't have the luxury of humans to work on
things that we once had, so I think as a community we should be
realistic about the benefits of consolidation and the downside to
continuing to have 2 different python SDKs.

Doing that implies the following:

  • Rework the underlying guts of openstacksdk to make it possible to
    replace shade's REST layer with openstacksdk. openstacksdk still doesn't
    have a 1.0 release, so we can break the few things we'll need to break.

  • Update the shade mission to indicate its purpose in life isn't just
    hiding deployer differences but rather is to provide a holistic
    cloud-centric (rather than service-centric) end-user API library.

  • Merge the two repos and retire one of them. Specifics on the mechanics
    of this below, but this will either result in moving the resource and
    service layer in openstacksdk into shade and adding appropriate
    attributes to the shade.OpenStackCloud object, or moving the
    shade.OpenStackCloud into something like openstack.cloud and making a
    shade backwards-compate shim. I lean towards the first, as we've been
    telling devs "use shade to talk to OpenStack" at hackathons and
    bootcamps and I'd rather avoid the messaging shift. However, pointing to
    an SDK called "The Python OpenStack SDK" and telling people to use it
    certainly has its benefits from a messaging perspective.

  • Collapse the core teams - members of the python-openstacksdk-core team
    who desire to stick around (I see Qiming doing reviews still, and Brian
    has been doing occasional ones even after his day-job shift) are welcome
    to be added to the shade-core team, but should not feel compelled to or
    like they'd be letting anyone down if they didn't. Day job priorities
    shift, it turns out.

Reworking the Guts


I did a scan through openstacksdk the other day to catalog what would
need to be reworked. The following are the big-ticket items:

  • drop stevedore/plugin support. An OpenStack REST client has no need
    for plugins. All services are welcome. note below

  • drop keystoneauth.Session subclass. It's over-riding things at the
    wrong layer. keystoneauth Adapter is the thing it wants to be.

  • stop using endpoint_filter in each Session call. Instead create an
    Adapter with the discovery parameters needed.

  • add support for per-request microversions. Based on the structure
    currently in openstacksdk once the wrapped Session has been replaced
    with keystoneauth Adapter this should work really nicely.

  • drop incomplete version discovery support in favor of the support in
    keystoneauth

  • drop Profile object completely and replace its use internally with the
    os-client-config CloudConfig object.

That's not a ton of work, TBH- I could probably do all of it in a single
long plane flight. It will break advanced users who might have been
using Profile (should be transparent to normal users) but as there is no
1.0 I think we should live with that. We might be able to make a shim
layer for the Profile interface to avoid breaking people using the
interface.

note on plugins

shade has a philosophy of not using plugins for service support that I'd
like to apply here. All OpenStack services are welcome to add code
directly. openstacksdk ALREADY contains code for tons of services. The
only thing plug-ability adds in this context is the ability to use
openstacksdk to support non-OpenStack services... and at this point I do
not think that is valuable. The only place this is currently used is in
the Profile anyway which allows defining an entrypoint to use to
override a service - and since I'm proposing we kill the Profile object
this all falls out as a matter of consequence.

Integrating with shade


The primary end-user concept in shade is an OpenStackCloud object, on
which one performs actions. The service that provides the action is
abstracted (this is done because actions such as 'list_images' may need
to be done on the image service or the compute service, depending on the
cloud). So a user does:

cloud = shade.openstackcloud(cloud='example')
images = cloud.list
images()

The primary end-user concept in openstacksdk is the Connection, which
has an object for each service. For example:

conn = openstack.connection.fromconfig(cloudname='example')
images = conn.image.images()

If we merge the two with the shade library being the primary interface,
we could add the current sdk service proxy objects as attributes to the
OpenStackCloud object, so that the following would work:

cloud = shade.openstackcloud(cloud='example')
images = cloud.list
images()
images = cloud.image.images()

If we did the merge the other way, we could either keep the Connection
concept and stitch the shade helpers on to it:

conn = openstack.connection.fromconfig(cloudname='example')
images = conn.list_images()
images = conn.image.images()

Or do a hybrid:

cloud = openstack.cloud(name='example')
images = cloud.list_images()
images = cloud.image.images()

If we go either of the routes of merging shade into openstacksdk then
the shade library itself could just be a simple sugar layer for
backwards compat that has things like:

def openstack_cloud(cloud=None, *args, **kwargs):
return openstack.cloud(name=cloud, *args, **kwargs)

and

class OpenStackCloud(openstack.cloud):
def init(self, cloud=None, *args, **kwargs):
super(OpenStackCloud, self).__init__(name=cloud, *args, **kwargs)

I kind of like the 'Connection' term, as it communicates that this is a
thing that has and shares a discrete remote connection (which is a
shared keystoneauth Session) OpenStackCloud in shade ACTUALLY
describes a cloud-region (as regions in OpenStack are essentially
independent clouds from an API consumption perspective. So I may be
leaning more towards merging in that direction.

  • data model - shade has a data model contract for the resources its
    knows about. This actually fits nicely with the Resource construct in
    openstacksdk, although there are some differences. We should likely push
    the data model and normalization contract into the openstacksdk resource
    layer so that people get matching resources regardless of whether they
    use the shade interop layer or the low-level per-service layer.

  • Implement some constructor smarts for easy pass-through of sdk service
    proxy methods into shade wrapper methods. For MANY of the remote calls,
    the list_ get_ search_ create_ update_ and delete_ methods are (or can
    be) mechanical passthrough from the SDK objects. We should be able to
    write some smart constructor logic that makes all the passthrough
    methods for us and just have explicit methods defined for the places
    where the shade layer legitimately needs to do a bunch of logic (like
    image upload and auto-ip support)

  • Make openstack.resource2.Resource a subclass of munch.Munch. This will
    be fun. If we want to be able to have model/normalization happen at the
    lower-level, we'd ultimately want the shade methods to be able to just
    return the object the sdk layer produces. Shade's interface defines that
    we return Munch objects (basically things that behave like dicts and
    objects) That's VERY similar to what the Resource already does - so if
    we subclass Munch the shade behavior will hold and the sdk coding style
    should also be able to hold as it is today. Otherwise we'd have to have
    every return in shade wrap the object in a
    munch.Munch(resource.to_dict()) which would lose information for when
    that object wants to be passed back in to the API later. (there are
    smarts in Resource for doing update things)

  • Migrate openstacksdk unit tests to use requestsmock. We're pretty
    much finished doing this in shade and it's been SUPER helpful.
    openstacksdk has a mock of the Session object in test
    proxybase2.py, so
    there's some good places to lay this in ... and as we merge the two
    obviously shade's requests
    mock unittests will continue to apply - but
    given the sdk's test organization I think we can get some really solid
    results by moving the mocking to the actual rest payload layer instead
    of mocking out the Session itself. It also is a great way to verify that
    things work as expected with varying payloads - so as a user finds an
    edge-case from a specific cloud, grabbing an http trace from them and
    constructing a specific test is a great way to deal with regressions.

We'll also need to make a specific rollout plan. shade has a strict
backwards-compat policy, so if we merge sdk into shade, it goes from
being 0.9 to being fully supported very quickly and we need to make sure
we don't have anything exposed in a public interface we don't want to
support for ages (resource -> resource2 should likely get finished and
then resource2 should likely get renamed back to resource before such a
release, for instance) If we merge the other direction and make the
current shade a backwards-compat shim lib, we'll also need to cut a 1.0
of sdk pretty quickly, as whatever passthrough object we are exposing
via the shade layer from sdk just adopted the shade backwards compat
contract. I don't have a detailed plan for this yet, but if we decide to
go down this path I'll make one.

Other TODO list items


It's not just all grunt work nobody can see. There's fun things to do too!

  • Add openstack.OpenStack or shade.AllClouds class - It's been a
    todo-list item for me for a while to make a wrapper class in shade that
    allows easy operations on a set of Clouds. Again, I like the semantics
    of openstack.OpenStack better - so that's another argument in favor of
    that direction of merge ... it would look something like this:

    make Connection objects for every cloud-region in clouds.yaml

    clouds = openstack.OpenStack()

    thread/asyncio parallel fetch of images across all clouds

    images = clouds.list_images()

    all objects in shade have a "location" field which contains

    cloud, region, domain and project info for the resource

    print([image for image in images if image.location.cloud='vexxhost'])

    Location is a required parameter for creation

    vexxhost = clouds.getlocation(name='vexxhost')
    clouds.create
    image(
    location=vexxhost, name='my-fedora-26',
    filename='fedora26.qcow2')

Most of this work can actually likely be done with one smart metaclass
... finding the methods on OpenStackCloud and either doing a set of
parallel gets on a list of objects or adding a location argument to
write operations doesn't vary depending on the type of object. Since all
the methods are named consistently, finding 'list*' and making a set of
corresponding list
methods on the OpenStack object is essentially just
one chunk of smart constructor.

  • Finish per-resource caching / batched client-side rate-limiting work.
    shade has a crazy cool ability to do batched and rate-limited operations
    ... this is how nodepool works at the scale it does. But it's currently
    only really plumbed through for server, floating-ip and port. (guess
    what nodepool has to deal with) This should be generalized to all of the
    resources, configurable on a per-resource name in clouds.yaml, and
    should work whether high or low level interfaces are used. This is super
    hard to get RIGHT, so it's one of those "spend 4 weeks writing 10 lines
    of code" kind of things, but it's also super important.

  • Implement a flag for toggling list/client-side-filter vs. remote-get
    operations. Most single-resource operations in shade are actually done
    as a list followed by a client-side filter. Again this model is there to
    support nodepool-scale (amusingly enough it makes it less load on the
    clouds at scale) but at small scale it is more costly and some users
    find it annoying. We've discussed having the ability to toggle this at
    constructor time - and then having things like the ansible modules
    default the flag to use remote-get instead of list/filter - since those
    are doing lots of independent processes so the optimization of
    list/filter isn't ever realized.

  • Implement smarter and more comprehensive "pushdown" filtering. I think
    we can piggyback a bunch of this off of the SDK layer - but there are
    attributes that can be used for server-side filtering, there are
    attributes that can't, and there are attributes that are client-side
    created via normalization that either can be translated into a
    server-side filter or must be client-side filtered. Resource has the
    structure for dealing with this sanely I believe, but it needs to be
    tracked through.

  • Add python3-style type annotations. We just started doing this in zuul
    and it's pretty cool - and is possible to do in a python2 compatible way.

Longer Goals


That gets us a home for openstacksdk, a path towards consolidation of
effort and a clear story for our end users. There are a few longer-term
things we should be keeping in mind as we work on this:

  • suitability for python-openstackclient. Dean and Steve have been
    laying in the groundwork for doing direct-REST in python-openstackclient
    because python-*client are a mess from an end-user perspective and
    openstacksdk isn't suitable. If we can sync on requirements hopefully we
    can produce something that python-openstackclient can honestly use for
    that layer instead of needing local code.

  • suitability for heat/horizon - both heat and horizon make calls to
    other OpenStack services as a primary operation (plenty of services make
    service-to-service calls, but for heat and horizon is a BIG part of
    their life) The results of this work should allow heat and horizon to
    remove local work they have using python-*client, doing local version
    discovery or any of the rest - and should expose to them rich primitives
    they can use easily.

Conclusion


As I mentioned at the top, I'd been thinking some of this already and
had planned on chatting with folks in person at the PTG, but it seems
we're at a place where that's potentially counter productive.

Depending on what people think I can follow this up with some governance
resolutions and more detailed specs.

Thanks!
Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Monty_Taylor (22,780 points)   2 4 7
0 votes

rage-quit

On Fri, Aug 4, 2017 at 11:52 AM, Monty Taylor mordred@inaugust.com wrote:

On 08/04/2017 03:24 AM, Thierry Carrez wrote:

Michael Johnson wrote:

I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have
also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended
in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend
development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who
the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks for raising this.

Beyond its limited activity, another issue is that it's not an official
project while its name make it a "default choice" for a lot of users
(hard to blame them for thinking that
http://git.openstack.org/cgit/openstack/python-openstacksdk is not the
official Python SDK for OpenStack, but I digress). So I agree that the
situation should be clarified.

I know that Monty has pretty strong feelings about this too, so I'll
wait for him to comment.

Oh boy. I'd kind of hoped we'd make it to the PTG before starting this
conversation ... guess not. :)

Concerns


I share the same concerns Thierry listed above. Specifically:

  • It is not an official project, but its name leads people to believe it's
    the "right" thing to use if they want to talk to OpenStack clouds using
    Python.

  • The core team is small to begin with, but recently got hit in a major
    way by shifts in company priorities.

I think we can all agree that those are concerns.

Three additional points:

  • The OpenStack AppDev group and the various appdev hackathons use shade,
    not openstacksdk. This is what we have people out in the world recommending
    people use when they write code that consumes OpenStack APIs. The Interop
    challenges at the Summits so far have all used Ansible's OpenStack modules,
    which are based on shade, because they were the thing that works.

  • Both shade and python-openstackclient have investigated using
    openstacksdk as their REST layer but were unable to because it puts some
    abstractions in layers that make it impossible to do some of the advanced
    things we need.

  • openstacksdk has internal implementations of things that exist at
    different points in the stack. We just added full support for version
    service and version discovery to keystoneauth, but openstacksdk has its own
    layer for that so it both can't use the ksa implementation and is not
    compliant with the API-WG consume guidelines.

It's not all bad! There is some great work in openstacksdk and it's a
shame there are some things that make it hard to consume. Brian, Qiming and
Terry have done a bunch of excellent work - and I'd like to not lose it to
the dustbin of corporate shifting interest.

warning - there is a very large text wall that follows. If you don't
care a ton on this topic, please stop reading now, otherwise you might
rage-quit computers altogether.

Proposal


I'd propose we have the shade team adopt the python-openstacksdk codebase.

This is obviously an aggressive suggestion and essentially represents a
takeover of a project. We don't have the luxury of humans to work on things
that we once had, so I think as a community we should be realistic about
the benefits of consolidation and the downside to continuing to have 2
different python SDKs.

Doing that implies the following:

  • Rework the underlying guts of openstacksdk to make it possible to
    replace shade's REST layer with openstacksdk. openstacksdk still doesn't
    have a 1.0 release, so we can break the few things we'll need to break.

  • Update the shade mission to indicate its purpose in life isn't just
    hiding deployer differences but rather is to provide a holistic
    cloud-centric (rather than service-centric) end-user API library.

  • Merge the two repos and retire one of them. Specifics on the mechanics
    of this below, but this will either result in moving the resource and
    service layer in openstacksdk into shade and adding appropriate attributes
    to the shade.OpenStackCloud object, or moving the shade.OpenStackCloud into
    something like openstack.cloud and making a shade backwards-compate shim. I
    lean towards the first, as we've been telling devs "use shade to talk to
    OpenStack" at hackathons and bootcamps and I'd rather avoid the messaging
    shift. However, pointing to an SDK called "The Python OpenStack SDK" and
    telling people to use it certainly has its benefits from a messaging
    perspective.

  • Collapse the core teams - members of the python-openstacksdk-core team
    who desire to stick around (I see Qiming doing reviews still, and Brian has
    been doing occasional ones even after his day-job shift) are welcome to be
    added to the shade-core team, but should not feel compelled to or like
    they'd be letting anyone down if they didn't. Day job priorities shift, it
    turns out.

Reworking the Guts


I did a scan through openstacksdk the other day to catalog what would need
to be reworked. The following are the big-ticket items:

  • drop stevedore/plugin support. An OpenStack REST client has no need for
    plugins. All services are welcome. note below

  • drop keystoneauth.Session subclass. It's over-riding things at the wrong
    layer. keystoneauth Adapter is the thing it wants to be.

  • stop using endpoint_filter in each Session call. Instead create an
    Adapter with the discovery parameters needed.

  • add support for per-request microversions. Based on the structure
    currently in openstacksdk once the wrapped Session has been replaced with
    keystoneauth Adapter this should work really nicely.

  • drop incomplete version discovery support in favor of the support in
    keystoneauth

  • drop Profile object completely and replace its use internally with the
    os-client-config CloudConfig object.

That's not a ton of work, TBH- I could probably do all of it in a single
long plane flight. It will break advanced users who might have been using
Profile (should be transparent to normal users) but as there is no 1.0 I
think we should live with that. We might be able to make a shim layer for
the Profile interface to avoid breaking people using the interface.

note on plugins

shade has a philosophy of not using plugins for service support that I'd
like to apply here. All OpenStack services are welcome to add code
directly. openstacksdk ALREADY contains code for tons of services. The only
thing plug-ability adds in this context is the ability to use openstacksdk
to support non-OpenStack services... and at this point I do not think that
is valuable. The only place this is currently used is in the Profile anyway
which allows defining an entrypoint to use to override a service - and
since I'm proposing we kill the Profile object this all falls out as a
matter of consequence.

Integrating with shade


The primary end-user concept in shade is an OpenStackCloud object, on
which one performs actions. The service that provides the action is
abstracted (this is done because actions such as 'list_images' may need to
be done on the image service or the compute service, depending on the
cloud). So a user does:

cloud = shade.openstackcloud(cloud='example')
images = cloud.list
images()

The primary end-user concept in openstacksdk is the Connection, which has
an object for each service. For example:

conn = openstack.connection.fromconfig(cloudname='example')
images = conn.image.images()

If we merge the two with the shade library being the primary interface, we
could add the current sdk service proxy objects as attributes to the
OpenStackCloud object, so that the following would work:

cloud = shade.openstackcloud(cloud='example')
images = cloud.list
images()
images = cloud.image.images()

If we did the merge the other way, we could either keep the Connection
concept and stitch the shade helpers on to it:

conn = openstack.connection.fromconfig(cloudname='example')
images = conn.list_images()
images = conn.image.images()

Or do a hybrid:

cloud = openstack.cloud(name='example')
images = cloud.list_images()
images = cloud.image.images()

If we go either of the routes of merging shade into openstacksdk then the
shade library itself could just be a simple sugar layer for backwards
compat that has things like:

def openstack_cloud(cloud=None, *args, **kwargs):
return openstack.cloud(name=cloud, *args, **kwargs)

and

class OpenStackCloud(openstack.cloud):
def init(self, cloud=None, *args, **kwargs):
super(OpenStackCloud, self).__init__(name=cloud, *args, **kwargs)

I kind of like the 'Connection' term, as it communicates that this is a
thing that has and shares a discrete remote connection (which is a shared
keystoneauth Session) OpenStackCloud in shade ACTUALLY describes a
cloud-region (as regions in OpenStack are essentially independent clouds
from an API consumption perspective. So I may be leaning more towards
merging in that direction.

  • data model - shade has a data model contract for the resources its knows
    about. This actually fits nicely with the Resource construct in
    openstacksdk, although there are some differences. We should likely push
    the data model and normalization contract into the openstacksdk resource
    layer so that people get matching resources regardless of whether they use
    the shade interop layer or the low-level per-service layer.

  • Implement some constructor smarts for easy pass-through of sdk service
    proxy methods into shade wrapper methods. For MANY of the remote calls, the
    list_ get_ search_ create_ update_ and delete_ methods are (or can be)
    mechanical passthrough from the SDK objects. We should be able to write
    some smart constructor logic that makes all the passthrough methods for us
    and just have explicit methods defined for the places where the shade layer
    legitimately needs to do a bunch of logic (like image upload and auto-ip
    support)

  • Make openstack.resource2.Resource a subclass of munch.Munch. This will
    be fun. If we want to be able to have model/normalization happen at the
    lower-level, we'd ultimately want the shade methods to be able to just
    return the object the sdk layer produces. Shade's interface defines that we
    return Munch objects (basically things that behave like dicts and objects)
    That's VERY similar to what the Resource already does - so if we subclass
    Munch the shade behavior will hold and the sdk coding style should also be
    able to hold as it is today. Otherwise we'd have to have every return in
    shade wrap the object in a munch.Munch(resource.to_dict()) which would
    lose information for when that object wants to be passed back in to the API
    later. (there are smarts in Resource for doing update things)

  • Migrate openstacksdk unit tests to use requestsmock. We're pretty much
    finished doing this in shade and it's been SUPER helpful. openstacksdk has
    a mock of the Session object in test
    proxybase2.py, so there's some good
    places to lay this in ... and as we merge the two obviously shade's
    requests
    mock unittests will continue to apply - but given the sdk's test
    organization I think we can get some really solid results by moving the
    mocking to the actual rest payload layer instead of mocking out the Session
    itself. It also is a great way to verify that things work as expected with
    varying payloads - so as a user finds an edge-case from a specific cloud,
    grabbing an http trace from them and constructing a specific test is a
    great way to deal with regressions.

We'll also need to make a specific rollout plan. shade has a strict
backwards-compat policy, so if we merge sdk into shade, it goes from being
0.9 to being fully supported very quickly and we need to make sure we don't
have anything exposed in a public interface we don't want to support for
ages (resource -> resource2 should likely get finished and then resource2
should likely get renamed back to resource before such a release, for
instance) If we merge the other direction and make the current shade a
backwards-compat shim lib, we'll also need to cut a 1.0 of sdk pretty
quickly, as whatever passthrough object we are exposing via the shade layer
from sdk just adopted the shade backwards compat contract. I don't have a
detailed plan for this yet, but if we decide to go down this path I'll make
one.

Other TODO list items


It's not just all grunt work nobody can see. There's fun things to do too!

  • Add openstack.OpenStack or shade.AllClouds class - It's been a todo-list
    item for me for a while to make a wrapper class in shade that allows easy
    operations on a set of Clouds. Again, I like the semantics of
    openstack.OpenStack better - so that's another argument in favor of that
    direction of merge ... it would look something like this:

    make Connection objects for every cloud-region in clouds.yaml

    clouds = openstack.OpenStack()

    thread/asyncio parallel fetch of images across all clouds

    images = clouds.list_images()

    all objects in shade have a "location" field which contains

    cloud, region, domain and project info for the resource

    print([image for image in images if image.location.cloud='vexxhost'])

    Location is a required parameter for creation

    vexxhost = clouds.getlocation(name='vexxhost')
    clouds.create
    image(
    location=vexxhost, name='my-fedora-26',
    filename='fedora26.qcow2')

Most of this work can actually likely be done with one smart metaclass ...
finding the methods on OpenStackCloud and either doing a set of parallel
gets on a list of objects or adding a location argument to write operations
doesn't vary depending on the type of object. Since all the methods are
named consistently, finding 'list*' and making a set of corresponding
list
methods on the OpenStack object is essentially just one chunk of
smart constructor.

  • Finish per-resource caching / batched client-side rate-limiting work.
    shade has a crazy cool ability to do batched and rate-limited operations
    ... this is how nodepool works at the scale it does. But it's currently
    only really plumbed through for server, floating-ip and port. (guess what
    nodepool has to deal with) This should be generalized to all of the
    resources, configurable on a per-resource name in clouds.yaml, and should
    work whether high or low level interfaces are used. This is super hard to
    get RIGHT, so it's one of those "spend 4 weeks writing 10 lines of code"
    kind of things, but it's also super important.

  • Implement a flag for toggling list/client-side-filter vs. remote-get
    operations. Most single-resource operations in shade are actually done as a
    list followed by a client-side filter. Again this model is there to support
    nodepool-scale (amusingly enough it makes it less load on the clouds at
    scale) but at small scale it is more costly and some users find it
    annoying. We've discussed having the ability to toggle this at constructor
    time - and then having things like the ansible modules default the flag to
    use remote-get instead of list/filter - since those are doing lots of
    independent processes so the optimization of list/filter isn't ever
    realized.

  • Implement smarter and more comprehensive "pushdown" filtering. I think
    we can piggyback a bunch of this off of the SDK layer - but there are
    attributes that can be used for server-side filtering, there are attributes
    that can't, and there are attributes that are client-side created via
    normalization that either can be translated into a server-side filter or
    must be client-side filtered. Resource has the structure for dealing with
    this sanely I believe, but it needs to be tracked through.

  • Add python3-style type annotations. We just started doing this in zuul
    and it's pretty cool - and is possible to do in a python2 compatible way.

Longer Goals


That gets us a home for openstacksdk, a path towards consolidation of
effort and a clear story for our end users. There are a few longer-term
things we should be keeping in mind as we work on this:

  • suitability for python-openstackclient. Dean and Steve have been laying
    in the groundwork for doing direct-REST in python-openstackclient because
    python-*client are a mess from an end-user perspective and openstacksdk
    isn't suitable. If we can sync on requirements hopefully we can produce
    something that python-openstackclient can honestly use for that layer
    instead of needing local code.

  • suitability for heat/horizon - both heat and horizon make calls to other
    OpenStack services as a primary operation (plenty of services make
    service-to-service calls, but for heat and horizon is a BIG part of their
    life) The results of this work should allow heat and horizon to remove
    local work they have using python-*client, doing local version discovery or
    any of the rest - and should expose to them rich primitives they can use
    easily.

Conclusion


As I mentioned at the top, I'd been thinking some of this already and had
planned on chatting with folks in person at the PTG, but it seems we're at
a place where that's potentially counter productive.

Depending on what people think I can follow this up with some governance
resolutions and more detailed specs.

Thanks!
Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kind regards,

Melvin Hillsman
mrhillsman@gmail.com
mobile: (832) 264-2646


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Melvin_Hillsman (4,480 points)   1 2 2
0 votes

Awesome Monty. This is a great proposal. I have no preference on which way these merge, but see huge value in straightening this out. Frankly I think some of the tempest plugin work could benefit from having an official and well maintained SDK as well.

So, I am in favor of getting the ball rolling here. I was really hoping to be able to use this for our dashboard work in Queens, but giving the work here I may need to do something in the interim. Maybe if these patches merge for python-openstacksdk I can use that for now and migrate as new SDK becomes available.

Michael

-----Original Message-----
From: Monty Taylor [mailto:mordred@inaugust.com]
Sent: Friday, August 4, 2017 9:53 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [python-openstacksdk] Status of python-openstacksdk project

Conclusion


As I mentioned at the top, I'd been thinking some of this already and had planned on chatting with folks in person at the PTG, but it seems we're at a place where that's potentially counter productive.

Depending on what people think I can follow this up with some governance resolutions and more detailed specs.

Thanks!
Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Michael_Johnson (4,380 points)   4 5
0 votes

Also note that this appears to exist:

https://github.com/openstack/python-openstackclient/blob/master/requirements.txt#L10

So even if python-openstacksdk is not a top level project, I would
assume that it being a requirement would imply that it is? Or perhaps
neither the python-openstackclient or python-openstacksdk should really
be used? I've been telling people that python-openstackclient should be
good to use (I hope that is still correct, though I do have to tell
people to not use python-openstackclient from python itself, and only
use it from bash/other shell).

Michael Johnson wrote:
Hi OpenStack developers,

I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks,
Michael (johnsom)

[1]
https://review.openstack.org/#/q/project:openstack/python-openstacksdk+statu
s:open+topic:%255Eoctavia.*


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by harlowja_at_fastmail (16,200 points)   2 5 6
0 votes

Monty,

  • drop stevedore/plugin support. An OpenStack REST client has no need for
    plugins. All services are welcome. note below

Back to 60s style of development? Just copy paste things? no plugins? no
architecture? no libs?

That's not going to work for dozens of OpenStack projects. It's just won't
scale. Every project should maintain plugin for their project. And it
should be enough to say "pip install python-client" that
extend the Core OpenStack python client and adds support of new commands.

The whole core part should be only about how to make plugins interface in
such way that it's easy to extend and provide to end user nice user
experience from both side (shell & python) and nice and stable interface
for client developers .

By the way stevedore is really providing very bad plugin experience and
should not be used definitely.

Best regards,
Boris Pavlovic

On Fri, Aug 4, 2017 at 12:05 PM, Joshua Harlow harlowja@fastmail.com
wrote:

Also note that this appears to exist:

https://github.com/openstack/python-openstackclient/blob/mas
ter/requirements.txt#L10

So even if python-openstacksdk is not a top level project, I would assume
that it being a requirement would imply that it is? Or perhaps neither the
python-openstackclient or python-openstacksdk should really be used? I've
been telling people that python-openstackclient should be good to use (I
hope that is still correct, though I do have to tell people to not use
python-openstackclient from python itself, and only use it from bash/other
shell).

Michael Johnson wrote:

Hi OpenStack developers,

I was wondering what is the current status of the python-openstacksdk
project. The Octavia team has posted some patches implementing our new
Octavia v2 API [1] in the SDK, but we have not had any reviews. I have
also
asked some questions in #openstack-sdks with no responses.
I see that there are some maintenance patches getting merged and a pypi
release was made 6/14/17 (though not through releases project). I'm not
seeing any mailing list traffic and the IRC meetings seem to have ended in
2016.

With all the recent contributor changes, I want to make sure the project
isn't adrift in the sea of OpenStack before we continue to spend
development
time implementing the SDK for Octavia. We were also planning to use it as
the backing for our dashboard project.

Since it's not in the governance projects list I couldn't determine who
the
PTL to ping would be, so I decided to ping the dev mailing list.

My questions:
1. Is this project abandoned?
2. Is there a plan to make it an official project?
3. Should we continue to develop for it?

Thanks,
Michael (johnsom)

[1]
https://review.openstack.org/#/q/project:openstack/python-op
enstacksdk+statu
s:open+topic:%255Eoctavia.*



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by boris_at_pavlovic.me (6,900 points)   1 4 6
0 votes

On Fri, Aug 4, 2017 at 11:52 AM, Monty Taylor mordred@inaugust.com wrote:
[...]
* Both shade and python-openstackclient have investigated using openstacksdk
as their REST layer but were unable to because it puts some abstractions in
layers that make it impossible to do some of the advanced things we need.

OSC does use the SDK for all Network API commands. That was a
mistake in the sense that we did it before SDK 1.0 was released and
still do not have 1.0.

  • openstacksdk has internal implementations of things that exist at
    different points in the stack. We just added full support for version
    service and version discovery to keystoneauth, but openstacksdk has its own
    layer for that so it both can't use the ksa implementation and is not
    compliant with the API-WG consume guidelines.

This was intended to make it free from dependencies, when first
concieved, none of these other libs existed.

I am coming around to believe that we need to slice a couple more
things up so they can be composed as needed rather then bite off the
entire thing in once piece.

[...]

I'd propose we have the shade team adopt the python-openstacksdk codebase.

++

This is obviously an aggressive suggestion and essentially represents a
takeover of a project. We don't have the luxury of humans to work on things
that we once had, so I think as a community we should be realistic about the
benefits of consolidation and the downside to continuing to have 2 different
python SDKs.

++

I thought it would be natural for OSC to adopt the SDK someday if
Brian did not get around to making it official, but the current
circumstances make it clear that we (OSC) do not have the resources to
do this. This proposal is much better and leads to a natural
coalescence of the parallel goals and projects.

Doing that implies the following:

  • Rework the underlying guts of openstacksdk to make it possible to replace
    shade's REST layer with openstacksdk. openstacksdk still doesn't have a 1.0
    release, so we can break the few things we'll need to break.

Sigh. OSC has been using the Network components of the SDK for a long
time in spite of SDK not being at 1.0. In retrospect that was a
mistake on my part but I believed at the time that 1.0 was close and
we had already ignored Network for far too long. We already have one
compatibility layer in the SDK due to the proxy refactor work that was
supposed to be the last thing before 1.0.

[...]

  • Merge the two repos and retire one of them. Specifics on the mechanics of
    this below, but this will either result in moving the resource and service
    layer in openstacksdk into shade and adding appropriate attributes to the
    shade.OpenStackCloud object, or moving the shade.OpenStackCloud into
    something like openstack.cloud and making a shade backwards-compate shim. I
    lean towards the first, as we've been telling devs "use shade to talk to
    OpenStack" at hackathons and bootcamps and I'd rather avoid the messaging
    shift. However, pointing to an SDK called "The Python OpenStack SDK" and
    telling people to use it certainly has its benefits from a messaging
    perspective.

I don't have a big concern about which repo is maintained. For OSC I
want what amounts to a low-level REST API, one example of which can be
found in openstackclient.api.*. Currently Shade is not quite the
right thing to back a CLI but now has the layer I want, and SDK does
not have that layer at all (it was proposed very early on and not
merged).

Is it better to have a single monolithic thing that has three
conceptual layers internally that can be individually consumed or to
have three things that get layered as needed?

  • drop keystoneauth.Session subclass. It's over-riding things at the wrong
    layer. keystoneauth Adapter is the thing it wants to be.

FWIW, OSC has this problem too...

  • suitability for python-openstackclient. Dean and Steve have been laying in
    the groundwork for doing direct-REST in python-openstackclient because
    python-*client are a mess from an end-user perspective and openstacksdk
    isn't suitable. If we can sync on requirements hopefully we can produce
    something that python-openstackclient can honestly use for that layer
    instead of needing local code.

As I mentioned, we already use the Networking portions of SDK, even
without a 1.0, and it has bit us already a couple of times. It has
long been my plan to convert to using the SDK, but that was when I
believed there would also be a lower-level API exposed that did not
require all of the application-level goodness and abstractions.

I personally feel like splitting out the low-level REST API layers
into a stand-alone piece that shade, SDK and OSC can all benefit from
would be our best course, but then I have been wrong about this
layering thing in the past, so I throw it out there to have something
that can be used to push against to get what everyone else seems to
want.

dt

--

Dean Troyer
dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Dean_Troyer (13,100 points)   1 3 3
0 votes

On Fri, 2017-08-04 at 12:26 -0700, Boris Pavlovic wrote:
By the way stevedore is really providing very bad plugin experience
and should not be used definitely. 

Perhaps entrypointer[1]? ;)

[1] https://pypi.python.org/pypi/entrypointer
--
Kevin L. Mitchell klmitch@mit.edu__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


responded Aug 4, 2017 by klmitch_at_mit.edu (600 points)  
0 votes

On Fri, Aug 4, 2017, at 02:37 PM, Kevin L. Mitchell wrote:
On Fri, 2017-08-04 at 12:26 -0700, Boris Pavlovic wrote:

By the way stevedore is really providing very bad plugin experience
and should not be used definitely. 

Perhaps entrypointer[1]? ;)

[1] https://pypi.python.org/pypi/entrypointer
--
Kevin L. Mitchell klmitch@mit.edu

The problems seem to be more with the use of entrypoints and the
incredible runtime cost for using them (which you've hinted at in
entrypointer's README). I don't think switching from $plugin lib to
$otherplugin lib changes much for tools like openstackclient unless we
first fix entrypoints (or avoid entrypoints all together). Until then
you must still rely on entrypoints to scan your python path which is
slow.

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On 08/04/2017 04:03 PM, Dean Troyer wrote:
On Fri, Aug 4, 2017 at 11:52 AM, Monty Taylor mordred@inaugust.com wrote:
[...]

  • Both shade and python-openstackclient have investigated using openstacksdk
    as their REST layer but were unable to because it puts some abstractions in
    layers that make it impossible to do some of the advanced things we need.

OSC does use the SDK for all Network API commands. That was a
mistake in the sense that we did it before SDK 1.0 was released and
still do not have 1.0.

Yah - although it got you out of the competing cliff issue, so that was
good.

  • openstacksdk has internal implementations of things that exist at
    different points in the stack. We just added full support for version
    service and version discovery to keystoneauth, but openstacksdk has its own
    layer for that so it both can't use the ksa implementation and is not
    compliant with the API-WG consume guidelines.

This was intended to make it free from dependencies, when first
concieved, none of these other libs existed.

Oh - totally. It made total sense at the time ... it's just the
surrounding context has changed.

I am coming around to believe that we need to slice a couple more
things up so they can be composed as needed rather then bite off the
entire thing in once piece.

[...]

I'd propose we have the shade team adopt the python-openstacksdk codebase.

++

This is obviously an aggressive suggestion and essentially represents a
takeover of a project. We don't have the luxury of humans to work on things
that we once had, so I think as a community we should be realistic about the
benefits of consolidation and the downside to continuing to have 2 different
python SDKs.

++

I thought it would be natural for OSC to adopt the SDK someday if
Brian did not get around to making it official, but the current
circumstances make it clear that we (OSC) do not have the resources to
do this. This proposal is much better and leads to a natural
coalescence of the parallel goals and projects.

Doing that implies the following:

  • Rework the underlying guts of openstacksdk to make it possible to replace
    shade's REST layer with openstacksdk. openstacksdk still doesn't have a 1.0
    release, so we can break the few things we'll need to break.

Sigh. OSC has been using the Network components of the SDK for a long
time in spite of SDK not being at 1.0. In retrospect that was a
mistake on my part but I believed at the time that 1.0 was close and
we had already ignored Network for far too long. We already have one
compatibility layer in the SDK due to the proxy refactor work that was
supposed to be the last thing before 1.0.

I don't think we need to break the things you're using, fwiw. In fact,
we can probably take it as a first-order requirement to not break OSC -
unless we find something SUPER intractable - and even then we should
talk about it first.

[...]

  • Merge the two repos and retire one of them. Specifics on the mechanics of
    this below, but this will either result in moving the resource and service
    layer in openstacksdk into shade and adding appropriate attributes to the
    shade.OpenStackCloud object, or moving the shade.OpenStackCloud into
    something like openstack.cloud and making a shade backwards-compate shim. I
    lean towards the first, as we've been telling devs "use shade to talk to
    OpenStack" at hackathons and bootcamps and I'd rather avoid the messaging
    shift. However, pointing to an SDK called "The Python OpenStack SDK" and
    telling people to use it certainly has its benefits from a messaging
    perspective.

I don't have a big concern about which repo is maintained. For OSC I
want what amounts to a low-level REST API, one example of which can be
found in openstackclient.api.*. Currently Shade is not quite the
right thing to back a CLI but now has the layer I want, and SDK does
not have that layer at all (it was proposed very early on and not
merged).

FWIW - now that we landed version discovery and microversion support in
keystoneauth - I'd say keystoneauth Adapter is actually now the
low-level REST API:

client = keystoneauth1.adapter.Adapter(
session=session,
servicetype='compute',
region
name='DFW',
minversion='2',
max
version='2.latest')
endpointdata = client.getendpointdata()
print(
"Microversion range is: ",
endpoint
data.minmicroversion,
endpoint
data.maxmicroversion)
# want 2.31 of servers
server
response = client.get('/servers', microversion='2.31')
server = serverresponse.json()['servers'][0]
# want 2.14 of keypairs
keypair
response = client.get('/keypairs', microversion='2.14')
# Don't care on volume attachments - don't specify one.
volumeattachmentsresponse = client.get(
'/servers/{id}/os-volume_attachments'.format(id=server['id']))

Is it better to have a single monolithic thing that has three
conceptual layers internally that can be individually consumed or to
have three things that get layered as needed?

It's a good question. I'm leaning to single repo for shade and sdk
mainly because since both do direct REST and dont' have other
dependencies, there's not really much difference from a composability
standpoint. If you just want the low-level SDK functions, use them. If
you want the hide-the-difference functions, use them - doesn't gain much
to have them split.

That said - if merging them is too much work, it's not super IMPORTANT
to merge them either.

  • drop keystoneauth.Session subclass. It's over-riding things at the wrong
    layer. keystoneauth Adapter is the thing it wants to be.

FWIW, OSC has this problem too...

Yah - I'm totally sorry I forgot to try to get your timing wrapper in to
ksa Session this pass so that you could drop that wrapper...

  • suitability for python-openstackclient. Dean and Steve have been laying in
    the groundwork for doing direct-REST in python-openstackclient because
    python-*client are a mess from an end-user perspective and openstacksdk
    isn't suitable. If we can sync on requirements hopefully we can produce
    something that python-openstackclient can honestly use for that layer
    instead of needing local code.

As I mentioned, we already use the Networking portions of SDK, even
without a 1.0, and it has bit us already a couple of times. It has
long been my plan to convert to using the SDK, but that was when I
believed there would also be a lower-level API exposed that did not
require all of the application-level goodness and abstractions.

I personally feel like splitting out the low-level REST API layers
into a stand-alone piece that shade, SDK and OSC can all benefit from
would be our best course, but then I have been wrong about this
layering thing in the past, so I throw it out there to have something
that can be used to push against to get what everyone else seems to
want.

Nod. I think that's ksa. We're doing almost nothing at the REST layer in
shade at this point - we're just using ksa Adapter. We have a wrapper
around it for two reasons:

  • TaskManager support for nodepool
  • translating ksa REST exceptions to shade exceptions for backwards compat

Then we do data normalization - but that's a 'hide cloud differences'
thing, not a thing that should be at that lower level.

rods actually just finished removing one additional thing we were doing,
which was stripping the top-level key from the results if there was one.
Turns out doing that really screws with pagination. whoops.

In any case, I agree with the need for a low-level REST interface.
Mostly since last we talked about it the missing pieces have hit ksa.

Looking at openstackclient.api.* though - we may be meaning different
things when we say "low level REST interface".

From the shade/sdk perspective, the extra metadata that's tracked for
each Resource object will, I believe make it easier to add basic new
CRUD for each resource type. I could certainly be VERY wrong about that
though.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 4, 2017 by Monty_Taylor (22,780 points)   2 4 7
...