settingsLogin | Registersettings

[openstack-dev] Endpoint structure: a free-for-all

0 votes

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

  2. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

  3. If 1 and 2 won't actually help me solve this, do you have any other
    suggestions that will? We already go left, right, and middle of each
    URI, so I'm out of directions to go, and we can't go back to the
    service catalog.

Thanks,

Brian


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 19, 2016 in openstack-dev by Brian_Curtin (1,420 points)   2
retagged Jan 26, 2017 by admin

17 Responses

0 votes

On 10/19/2016 05:32 PM, Brian Curtin wrote:
I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the
endpoints for all services in the cloud. Within each service,
determining the (micro)version of the API is unfortunately going to be a
per-service endeavour. For some APIs, a microversion header is returned,
others don't have microversions. The microversion header is
unfortunately not standardized for all APIs that use microversions,
though a number of us would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement REST
API, for what it's worth.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint
stuff? If so, my preference has been that such "internal vs. external"
routing should be handled via the Keystone service catalog returning a
set of endpoints depending on the source (or X-forwarded-for) IP. So,
requests from "internal" networks (for whatever definition of "internal"
you want) return a set of endpoint URLs reflecting the "internal" endpoints.

  1. If 1 and 2 won't actually help me solve this, do you have any other
    suggestions that will? We already go left, right, and middle of each
    URI, so I'm out of directions to go, and we can't go back to the
    service catalog.

I really don't understand why the service catalog should not be the
thing that we use to catalog the... services. To me it seems obvious the
Keystone service catalog is the focal point for what's needed here.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Jay_Pipes (59,760 points)   3 11 14
0 votes

On 10/19/2016 10:32 AM, Brian Curtin wrote:
I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

  2. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

  3. If 1 and 2 won't actually help me solve this, do you have any other
    suggestions that will? We already go left, right, and middle of each
    URI, so I'm out of directions to go, and we can't go back to the
    service catalog.

Thanks,

Brian


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

That's a tricky one. Just yesterday I was looking into how Tempest
creates the service clients it uses and lists versions, e.g. for compute:

https://github.com/openstack/tempest/blob/13.0.0/tempest/lib/services/compute/versions_client.py#L27

I was trying to figure out where that base_url value came from and in
the case of Tempest it's from an auth provider class, and I think the
services inside that thing are created from this code at some point:

https://github.com/openstack/tempest/blob/13.0.0/tempest/config.py#L1405

So at a basic level that builds the client with config options for the
service type (e.g. compute) and endpoint type (e.g. publicURL), so then
it can lookup the publicURL for the 'compute' service/endpoint in the
service catalog and then start parsing the endpoint URL to do a GET on
the root endpoint for compute to get the available API versions.

It doesn't sound like the same case with the SDK though since you don't
have a config file telling you what exact things to lookup in the
service catalog...which seems like the main issue.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Matt_Riedemann (48,320 points)   3 10 25
0 votes

On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes jaypipes@gmail.com wrote:
On 10/19/2016 05:32 PM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number of us
would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement REST
API, for what it's worth.

I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.

The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint stuff? If
so, my preference has been that such "internal vs. external" routing should
be handled via the Keystone service catalog returning a set of endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you want) return
a set of endpoint URLs reflecting the "internal" endpoints.

This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.

I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.

  1. If 1 and 2 won't actually help me solve this, do you have any other
    suggestions that will? We already go left, right, and middle of each
    URI, so I'm out of directions to go, and we can't go back to the
    service catalog.

I really don't understand why the service catalog should not be the thing
that we use to catalog the... services. To me it seems obvious the Keystone
service catalog is the focal point for what's needed here.

It is the thing that catalogs the services, but it's currently a mixed
bag. I'm straddling the line between "it has what I need in all cases
for a service" and "it has a pointer to what I need for this service"
by parsing out URLs and reconstructing them, acting as if the latter
situation is already here. This would actually be ok if it weren't for
the URL structure being all over the map, potentially pointing me in
the wrong place.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Brian_Curtin (1,420 points)   2
0 votes

On 10/19/2016 01:40 PM, Brian Curtin wrote:
On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes jaypipes@gmail.com wrote:

On 10/19/2016 05:32 PM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number of us
would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement REST
API, for what it's worth.

I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.

The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint stuff? If
so, my preference has been that such "internal vs. external" routing should
be handled via the Keystone service catalog returning a set of endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you want) return
a set of endpoint URLs reflecting the "internal" endpoints.

This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.

I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.

We should really unwind where you got that information from. The point
of the service catalog is to be exactly what you want it to be. If you,
as a writer of consuming software, have found problems with it's schema,
or the way it's commonly implemented, that make it fail for that
purpose, please be specific about those.

For instance, there was a long standing feeling that major versions
should not be listed in the catalog. However, that was mostly from a
purity perspective than a real world one. And at the end of the day we
definitely don't want to make every SDK build a heuristic for guessing
where endpoints are. If listing major versions is key, lets take that as
feedback and make the schema changes to do that.

We had a bit of an effort to work on this 2 cycles ago, but it got back
burnered in getting the api-ref work sorted (which is sort of a prereq),
but as that is finalizing this is something we can and should jump back
into.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Sean_Dague (66,200 points)   4 11 18
0 votes

On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague sean@dague.net wrote:
On 10/19/2016 01:40 PM, Brian Curtin wrote:

On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes jaypipes@gmail.com wrote:

On 10/19/2016 05:32 PM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number of us
would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement REST
API, for what it's worth.

I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.

The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint stuff? If
so, my preference has been that such "internal vs. external" routing should
be handled via the Keystone service catalog returning a set of endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you want) return
a set of endpoint URLs reflecting the "internal" endpoints.

This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.

I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.

We should really unwind where you got that information from. The point
of the service catalog is to be exactly what you want it to be. If you,
as a writer of consuming software, have found problems with it's schema,
or the way it's commonly implemented, that make it fail for that
purpose, please be specific about those.

For instance, there was a long standing feeling that major versions
should not be listed in the catalog. However, that was mostly from a
purity perspective than a real world one. And at the end of the day we
definitely don't want to make every SDK build a heuristic for guessing
where endpoints are. If listing major versions is key, lets take that as
feedback and make the schema changes to do that.

We had a bit of an effort to work on this 2 cycles ago, but it got back
burnered in getting the api-ref work sorted (which is sort of a prereq),
but as that is finalizing this is something we can and should jump back
into.

This started back in August when it came up that we didn't know where
that Keystone v3 endpoint was. After talking with a few people, Steve
Martinelli mentioned that at least as of then, hitting the unversioned
endpoint was the way to solve that. It being unlisted anywhere was
something for me to figure out (via that path manipulation from the
given v2), but it was later mentioned that ideally they would like to
have unversioned endpoints in the catalog anyway. I'm talking to Steve
now and perhaps I took that too far in extrapolating which direction
things were going in reality, but it was a solution that had to be
undertaken nonetheless and was seen as the best way forward at the
time. It's also the only one that mostly works at the moment.

In the end, I'll take listing major versions as long as it's accurate
and complete, but I'll also take listing the service root even if it
means an extra request for me to determine those versions.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Brian_Curtin (1,420 points)   2
0 votes

On 10/19/2016 2:27 PM, Brian Curtin wrote:
On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague sean@dague.net wrote:

On 10/19/2016 01:40 PM, Brian Curtin wrote:

On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes jaypipes@gmail.com wrote:

On 10/19/2016 05:32 PM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number of us
would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement REST
API, for what it's worth.

I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.

The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint stuff? If
so, my preference has been that such "internal vs. external" routing should
be handled via the Keystone service catalog returning a set of endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you want) return
a set of endpoint URLs reflecting the "internal" endpoints.

This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.

I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.

We should really unwind where you got that information from. The point
of the service catalog is to be exactly what you want it to be. If you,
as a writer of consuming software, have found problems with it's schema,
or the way it's commonly implemented, that make it fail for that
purpose, please be specific about those.

For instance, there was a long standing feeling that major versions
should not be listed in the catalog. However, that was mostly from a
purity perspective than a real world one. And at the end of the day we
definitely don't want to make every SDK build a heuristic for guessing
where endpoints are. If listing major versions is key, lets take that as
feedback and make the schema changes to do that.

We had a bit of an effort to work on this 2 cycles ago, but it got back
burnered in getting the api-ref work sorted (which is sort of a prereq),
but as that is finalizing this is something we can and should jump back
into.

This started back in August when it came up that we didn't know where
that Keystone v3 endpoint was. After talking with a few people, Steve
Martinelli mentioned that at least as of then, hitting the unversioned
endpoint was the way to solve that. It being unlisted anywhere was
something for me to figure out (via that path manipulation from the
given v2), but it was later mentioned that ideally they would like to
have unversioned endpoints in the catalog anyway. I'm talking to Steve
now and perhaps I took that too far in extrapolating which direction
things were going in reality, but it was a solution that had to be
undertaken nonetheless and was seen as the best way forward at the
time. It's also the only one that mostly works at the moment.

In the end, I'll take listing major versions as long as it's accurate
and complete, but I'll also take listing the service root even if it
means an extra request for me to determine those versions.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I personally thought long-term we wanted unversioned endpoints in the
service catalog, and if you want to do version discovery, you do a GET
on a particular service's endpoint URL in the service catalog, and that
returns the list of available API versions for that service along with a
status value (CURRENT, SUPPORTED, DEPRECATED) and any microversion
ranges within those.

I know we have 3 versions of keystone in the service catalog, which I
find pretty nasty personally, plus the fact that a lot of the client
code we have has had to burn 'volumev2' in as a default endpoint_type to
use cinder. IMO the service catalog should just have a 'volume' endpoint
and if I want to know the supported versions (v1, v2 and/or v3), I do a
GET on the publicURL for the volume endpoint from the service catalog.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Matt_Riedemann (48,320 points)   3 10 25
0 votes

On 10/19/2016 3:22 PM, Matt Riedemann wrote:
On 10/19/2016 2:27 PM, Brian Curtin wrote:

On Wed, Oct 19, 2016 at 2:03 PM, Sean Dague sean@dague.net wrote:

On 10/19/2016 01:40 PM, Brian Curtin wrote:

On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes jaypipes@gmail.com wrote:

On 10/19/2016 05:32 PM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured,
and I
think we're out of ways to approach it that catch all of the ways
that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less
services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get
E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we
chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be
solved.

  1. Is there any reliable, functional, and accurate programmatic
    way to
    get the versions and endpoints that all services on a cloud offer?

The Keystone service catalog should be the thing that provides the
endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others
don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number
of us
would like to see a single:

OpenStack-API-Version: , ...

header supported. This is the header supported in the new placement
REST
API, for what it's worth.

I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.

The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.

  1. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them,
    not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

I presume you are referring to the "internal" vs "public" endpoint
stuff? If
so, my preference has been that such "internal vs. external"
routing should
be handled via the Keystone service catalog returning a set of
endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you
want) return
a set of endpoint URLs reflecting the "internal" endpoints.

This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.

I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.

We should really unwind where you got that information from. The point
of the service catalog is to be exactly what you want it to be. If you,
as a writer of consuming software, have found problems with it's schema,
or the way it's commonly implemented, that make it fail for that
purpose, please be specific about those.

For instance, there was a long standing feeling that major versions
should not be listed in the catalog. However, that was mostly from a
purity perspective than a real world one. And at the end of the day we
definitely don't want to make every SDK build a heuristic for guessing
where endpoints are. If listing major versions is key, lets take that as
feedback and make the schema changes to do that.

We had a bit of an effort to work on this 2 cycles ago, but it got back
burnered in getting the api-ref work sorted (which is sort of a prereq),
but as that is finalizing this is something we can and should jump back
into.

This started back in August when it came up that we didn't know where
that Keystone v3 endpoint was. After talking with a few people, Steve
Martinelli mentioned that at least as of then, hitting the unversioned
endpoint was the way to solve that. It being unlisted anywhere was
something for me to figure out (via that path manipulation from the
given v2), but it was later mentioned that ideally they would like to
have unversioned endpoints in the catalog anyway. I'm talking to Steve
now and perhaps I took that too far in extrapolating which direction
things were going in reality, but it was a solution that had to be
undertaken nonetheless and was seen as the best way forward at the
time. It's also the only one that mostly works at the moment.

In the end, I'll take listing major versions as long as it's accurate
and complete, but I'll also take listing the service root even if it
means an extra request for me to determine those versions.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I personally thought long-term we wanted unversioned endpoints in the
service catalog, and if you want to do version discovery, you do a GET
on a particular service's endpoint URL in the service catalog, and that
returns the list of available API versions for that service along with a
status value (CURRENT, SUPPORTED, DEPRECATED) and any microversion
ranges within those.

I know we have 3 versions of keystone in the service catalog, which I
find pretty nasty personally, plus the fact that a lot of the client
code we have has had to burn 'volumev2' in as a default endpoint_type to
use cinder. IMO the service catalog should just have a 'volume' endpoint
and if I want to know the supported versions (v1, v2 and/or v3), I do a
GET on the publicURL for the volume endpoint from the service catalog.

Sorry, correction, I know we have 3 versions of cinder in the service
catalog...

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Matt_Riedemann (48,320 points)   3 10 25
0 votes

On Wed, Oct 19, 2016 at 2:27 PM, Brian Curtin brian@python.org wrote:
...

This started back in August when it came up that we didn't know where
that Keystone v3 endpoint was. After talking with a few people, Steve
Martinelli mentioned that at least as of then, hitting the unversioned
endpoint was the way to solve that. It being unlisted anywhere was
something for me to figure out (via that path manipulation from the
given v2), but it was later mentioned that ideally they would like to
have unversioned endpoints in the catalog anyway.

devstack now sets up the identity/keystone endpoints as unversioned. So you
get an endpoint with "http://192.168.122.102:5000" for example. So this is
what we're testing with now and you're lucky if a versioned endpoint works
at all ;).

I'm talking to Steve
now and perhaps I took that too far in extrapolating which direction
things were going in reality, but it was a solution that had to be
undertaken nonetheless and was seen as the best way forward at the
time. It's also the only one that mostly works at the moment.

In the end, I'll take listing major versions as long as it's accurate
and complete, but I'll also take listing the service root even if it
means an extra request for me to determine those versions.

--
- Brant


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2016 by Brant_Knudson (5,640 points)   1 2 2
0 votes

On 10/19/2016 04:22 PM, Matt Riedemann wrote:
I personally thought long-term we wanted unversioned endpoints in the
service catalog, and if you want to do version discovery, you do a GET
on a particular service's endpoint URL in the service catalog, and that
returns the list of available API versions for that service along with a
status value (CURRENT, SUPPORTED, DEPRECATED) and any microversion
ranges within those.

I know we have 3 versions of keystone in the service catalog, which I
find pretty nasty personally, plus the fact that a lot of the client
code we have has had to burn 'volumev2' in as a default endpoint_type to
use cinder. IMO the service catalog should just have a 'volume' endpoint
and if I want to know the supported versions (v1, v2 and/or v3), I do a
GET on the publicURL for the volume endpoint from the service catalog.

At 5 services, maybe. But at 50+ services (and growing) I think that the
idea of get an endpoint, then have custom parsing code for every service
because their version documents are different, is a really bad UX.

The reason we have volume, volumev2, and volumev3 is that no one
actually wants the unversioned volume endpoint. You can't do anything
with it. Everyone wants the actual endpoint that has resources.

We can solve this for all consumers by adding additional version field
to the catalog. This was the direction we were headed last spring before
the api-ref work took over.

I think Brian's initial complaints (which are very valid) here really
point to the fact that punting on that puts SDK and client authors in a
place where they are going to end up writing a ton of heuristic code to
guess url structure. Which would be sad for all of us. :(

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2016 by Sean_Dague (66,200 points)   4 11 18
0 votes

On 10/19/2016 12:07 PM, Matt Riedemann wrote:
On 10/19/2016 10:32 AM, Brian Curtin wrote:

I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.

In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.

We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports

Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.

However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:

E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice

In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
reversed()' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.

So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?

This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.

Ultimately, I would like to know how something like this can be solved.

  1. Is there any reliable, functional, and accurate programmatic way to
    get the versions and endpoints that all services on a cloud offer?

  2. Are there any guidelines, rules, expectations, or other
    documentation on how services can be installed and their endpoints
    structured that are helpful to people build apps that use them, not in
    those trying to install and operate them? I've looked around a few
    times and found nothing useful. A lot of what I've found has
    referenced suggestions for operators setting them up behind various
    load balancing tools.

  3. If 1 and 2 won't actually help me solve this, do you have any other
    suggestions that will? We already go left, right, and middle of each
    URI, so I'm out of directions to go, and we can't go back to the
    service catalog.

Thanks,

Brian


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

That's a tricky one. Just yesterday I was looking into how Tempest
creates the service clients it uses and lists versions, e.g. for compute:

https://github.com/openstack/tempest/blob/13.0.0/tempest/lib/services/compute/versions_client.py#L27

I was trying to figure out where that base_url value came from and in
the case of Tempest it's from an auth provider class, and I think the
services inside that thing are created from this code at some point:

https://github.com/openstack/tempest/blob/13.0.0/tempest/config.py#L1405

So at a basic level that builds the client with config options for the
service type (e.g. compute) and endpoint type (e.g. publicURL), so then
it can lookup the publicURL for the 'compute' service/endpoint in the
service catalog and then start parsing the endpoint URL to do a GET on
the root endpoint for compute to get the available API versions.

It doesn't sound like the same case with the SDK though since you don't
have a config file telling you what exact things to lookup in the
service catalog...which seems like the main issue.

In os-client-config, we defer all of this to keystoneauth1. Basically,
we collect servicetype, endpointtype and a pile of other parameters
(service_name, cert, key, favorite color) and pass them to a few
different keystoneauth calls to wind up with a
keystoneauth1.adapter.Adapter, which is a thing which behaves like a
requests Session but is rooted on a particular endpoint from the catalog
by name and knows about tokens.

For instance:

client = osclientconfig.makerestclient('compute', cloud='vexxhost')

Gets you one of those. With it, you can do:

client.get('/servers')

Nova doesn't have more than one major version in reality, so the /
endpoint isnt' a thing for nova.

for glance:

client = osclientconfig.makerestclient('image', cloud='vexxhost')
client.get('/v1/images')

or

client.get('/v2/images')

you can, with glance, typically do:

client.get('/')

and get the version endpoint.

In case you're wondering, the output of that last line is:

{u'versions': [{u'status': u'CURRENT', u'id': u'v2.3', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v2/', u'rel':
u'self'}]}, {u'status': u'SUPPORTED', u'id': u'v2.2', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v2/', u'rel':
u'self'}]}, {u'status': u'SUPPORTED', u'id': u'v2.1', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v2/', u'rel':
u'self'}]}, {u'status': u'SUPPORTED', u'id': u'v2.0', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v2/', u'rel':
u'self'}]}, {u'status': u'SUPPORTED', u'id': u'v1.1', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v1/', u'rel':
u'self'}]}, {u'status': u'SUPPORTED', u'id': u'v1.0', u'links':
[{u'href': u'http://image-ca-ymq-1.vexxhost.net/v1/', u'rel': u'self'}]}]}

(yes, all of these lines are live)

Similar with neutron:

client = osclientconfig.makerestclient('network', cloud='vexxhost')
client.get('/').json()
{u'versions': [{u'status': u'CURRENT', u'id': u'v2.0', u'links':
[{u'href': u'http://network-ca-ymq-1.vexxhost.net/v2.0', u'rel':
u'self'}]}]}

Volumes is more like nova:

client = osclientconfig.makerestclient('volume', cloud='vexxhost')
client.get('/volumes').json()
{u'volumes': []}

with get('/') throwing a 404.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2016 by Monty_Taylor (22,780 points)   2 5 8
...