settingsLogin | Registersettings

[openstack-dev] [api] [Nova] [Ironic] [Magnum] Microversion guideline in API-WG

0 votes

Hi, guys,

I'm working on adding Microversion into the API-WG's guideline which make sure we have consistent Microversion behavior in the API for user.
The Nova and Ironic already have Microversion implementation, and as I know Magnum https://review.openstack.org/#/c/184975/ is going to implement Microversion also.

Hope all the projects which support( or plan to) Microversion can join the review of guideline.

The Mircoversion specification(this almost copy from nova-specs): https://review.openstack.org/#/c/187112
And another guideline for when we should bump Mircoversion https://review.openstack.org/#/c/187896/

As I know, there already have a little different between Nova and Ironic's implementation. Ironic return min/max version when the requested
version doesn't support in server by http-headers. There isn't such thing in nova. But that is something for version negotiation we need for nova also.
Sean have pointed out we should use response body instead of http headers, the body can includes error message. Really hope ironic team can take a
look at if you guys have compelling reason for using http headers.

And if we think return body instead of http headers, we probably need think about back-compatible also. Because Microversion itself isn't versioned.
So I think we should keep those header for a while, does make sense?

Hope we have good guideline for Microversion, because we only can change Mircoversion itself by back-compatible way.

Thanks
Alex Xu


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 4, 2015 in openstack-dev by Xu,_Hejie (1,160 points)   2

45 Responses

0 votes

On Sat, Jun 20, 2015 at 9:14 AM, Devananda van der Veen
devananda.vdv@gmail.com wrote:
Almost all of our discussions so far on this topic have left something out,
which Monty pointed out to me last week. I'm following up now because
E_TRAVEL...

tldr;
What we're versioning here are API's, not packages. It's not a question of
numbering and dependency ordering, but of communicating support[ed|able]
interfaces between running services. Libtool is more relevant than semver.

http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html

Right now we lack a means to express that the API response is
"compatible-with" a particular previously-released version of the API. If we
had that, instead of "current-version", I believe we would have much happier
users (and a happier Dmitry and jroll).

Long version...
Every HTTP response from Ironic today includes three headers: min, max, and
version. The service can present an older API version, as long as it is
greater-than-or-equal-to the minimum supported version, even if that version
is incompatible with the maximum supported version. It does this by
rewriting responses to match what was expected in the requested (older)
version.

When the newer version is identical for all interfaces present in the
older version, this can be called compatible. Dmitry's point is that we
don't need to hide newer interfaces from users who request an older API
version, because the client won't know or care about things that weren't in
the version it requested.

However, we do need to signal their presence, and we don't have a good
means for that right now. We also need to signal to the client that the
response given is "compatible with" a certain "age" of API, even if it's not
exactly the same. And we don't have any means for that, either.

Time for an example....

Let's say that an incompatible change was made in v1.3. Let's also say that
a change was made in v1.5 that added a new endpoint. Today, this is what the
response headers would look like when calling a server running v1.5.

a) client requests v1.2, receives headers (min: 1.1, max: 1.5, current: 1.2)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5, current 1.4)
c) client requests v1.5, receives headers (min: 1.1, max: 1.5, current: 1.5)

What we have implemented today is that in case (b), the service will hide
any changes that we introduced in v1.5. But those changes did not affect any
functionality of the v1.4 API, so Dmitry objects to this. So do I.

The issue at hand is the response in case (b) ... though after spending the
last several months working on api versioning, I actually think all of those
are poor responses.

What I believe we should have:
a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.1)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)
b) client requests v1.5, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)

This is nice idea to return "compatible-with" information. But I feel
each microversion either backward compatible or not should have their
unique output only for what they had been introduced (not include new
version changes).

If older version shows what is in newer version(backward comp) then,
there is no meaning
of introducing those compatible changes as microversion. And if so
then it blank out whole
idea of publishing changes/features through microversion even they are
backward compatible.

How about combining those means older version do not include anything
from newer version
but it returns "compatible-with:" information also which can be useful
for user to bump version easily.

Yes -- (b) and (c) are identical responses.

Discuss.

-Devananda

On Tue, Jun 16, 2015 at 7:13 AM Dmitry Tantsur dtantsur@redhat.com wrote:

On 06/16/2015 03:47 PM, Jim Rollenhagen wrote:

On Tue, Jun 16, 2015 at 08:56:37AM +0200, Dmitry Tantsur wrote:

On 06/04/2015 08:58 AM, Xu, Hejie wrote:

Hi, guys,
I’m working on adding Microversion into the API-WG’s guideline which
make sure we have consistent Microversion behavior in the API for
user.
The Nova and Ironic already have Microversion implementation, and as I
know Magnum _https://review.openstack.org/#/c/184975/_ is going to
implement Microversion also.
Hope all the projects which support( or plan to) Microversion can join
the review of guideline.
The Mircoversion specification(this almost copy from nova-specs):
_https://review.openstack.org/#/c/187112_
And another guideline for when we should bump Mircoversion
_https://review.openstack.org/#/c/187896/_
As I know, there already have a little different between Nova and
Ironic’s implementation. Ironic return min/max version when the
requested
version doesn’t support in server by http-headers. There isn’t such
thing in nova. But that is something for version negotiation we need
for
nova also.
Sean have pointed out we should use response body instead of http
headers, the body can includes error message. Really hope ironic team
can take a
look at if you guys have compelling reason for using http headers.
And if we think return body instead of http headers, we probably need
think about back-compatible also. Because Microversion itself isn’t
versioned.
So I think we should keep those header for a while, does make sense?
Hope we have good guideline for Microversion, because we only can
change
Mircoversion itself by back-compatible way.
Thanks
Alex Xu

Hi all!

I'd like to try put in feedback based on living with microversions in
Kilo
release of Ironic.

And here's my take, based on my experiences. Keep in mind I'm a core
reviewer, a developer, and an operator of Ironic.

Thanks Jim, much appreciated!

From an ops perspective, our team has built our fair share of tooling
to
help us run Ironic. Some of it uses the REST API via python or node.js,
and of course we all use the CLI client often.

We also continuously deploy Ironic, for full transparency. My experience
is not with how this works every 6 months, but in the day-to-day.

First of all, after talking to folks off-list, I realized that we all,
and
the spec itself, confuse 3 aspects of microversion usage:

  1. protecting from breaking changes.
    This is clearly a big win from user's point of view, and it allowed us
    to
    conduct painful change with renaming an important node state in our
    state
    machine. It will allows us even worse change this cycle: change of the
    default state.

+1. Good stuff. My tooling doesn't break when I upgrade. Yay.

  1. API discoverability.
    While I believe that there maybe be better implementation of this idea,
    I
    think I got it now. People want services to report API versions they
    support. People want to be able to request a specific version, and fail
    early if it is not present. Also +1 from me.

I don't tend to personally do this. I usually am aware of what version
of Ironic I'm running against. However I see how this could be useful
for other folks.

I do, however, use the versions to say, "Oh, I can now request 1.5 which
has logical names! That's useful, let's set those to the names in our
CMDB." Now my tooling that interacts with the CMDB and Ironic can look
at the version and decide to use node.name instead of the old hack we
used to use.

  1. hiding new features from older clients
    This is not directly stated by the spec, but many people imply it, and
    Nova
    and Ironic did it in Kilo. I want us to be clear: it is not the same as

2.

You can report versions, but still allow new features to be used.

This is still totally useful. If you know what version you are running
against, you know exactly what features are available.

"You know" is about #2 - that's where confusion is :)
so if you know, that moving to inspection state is disallowed for your
tooling (but not for the whole system!), what does it give you?

I think the disconnect here is that we don't expect users (whether those
are people or computers) to explicitly request a version. We need to
message better that if you are using Ironic or building a tool against
Ironic's API, you should be pinning the version. We also need to take
this comment block[0] and put it in our docs, so users know what each
version does.

Knowing that I get feature X when I upgrade to version Y is useful.

It is this particular thing that gets -2 from me, after I've seen how
it
worked in practice, and that's why.

First of all, I don't believe anyone needs it. Seriously, I can't
imagine a
user asking "please prevent me from using non-breaking changes". And
attempt
to implement it was IMO a big failure for the following reasons:

a) It's hard to do. Even we, the core team, got confused, and for
non-core
people it took several iteration to do right. It's a big load on both
developers and reviewers.

I do agree with this. It's been painful. However, I think we're mostly
past that pain at this point. Does this patch[1] look like developer
pain?

It's not that painful to write. Now. When we have 10-20 version, it
probably will :)
anyway, it's hard to explain newcomers how to do it, and it's hard to
review the result. we failed at it, e.g. with error codes.

b) It's incomplete (at least for Ironic). We have several API-exposed
things
that are just impossible to hide. Good example are node states: if node
is
in a new state, we can't but expose it to older tooling. Our free-form
JSON
fields properties, instanceinfo, driverinfo and driverinternalinfo
are
examples as well. It's useless to speak about API contract, while we
have
those.

I somewhat agree here.

With node states, there are cases where we were able to hide it
(NOSTATE -> AVAILABLE), and cases where we were not (adding MANAGEABLE).
However, this list of states is (AIUI) not part of the API contract;
rather the verbs available to move between states are.

What's the point in contract, if there are things not covered by it that
drastically change the system behavior?

As far as JSON fields, we've never had a contract around what keys are
available. Only the semantics of working with those fields, and which
fields exist.

ditto as above: you can request new features by modifying driver_info.

c) It gives additional push back to making (required) breaking changes.
We
already got suggestions to have ONE MORE feature gating for breaking
changes. Reason: people will need to increase microversions to get
features,
and your breaking change will prevent it.

This is just silly. If 1.10 breaks a user, and the user wants 1.11,
they'll need to fix that breakage.

++ but not everyone agreed on the summit, when I was talking about
ENROLL state

d) It requires a hard compromise on the CLI tool. You either default it
to
1.0 forever, and force all the people to get used to figuring out
version
numbers and using ironic --ironic-api-version x.y every time
(terrible
user experience), or you default it to some known good version, bumping
it
from time to time. This, in turn, has 2 more serious problems:

I disagree that pinning a version all the time is a terrible experience.
We already require a number of options for authentication (OSUSERNAME,
OS
PASSWORD, etc etc). How many folks do you think type these in every
time? Solution is simple: add IRONICAPIVERSION to whatever exports the
other environment variables.

It's not that bad, especially if devstack/tripleo will provide some
reasonable default for you.

I remember, however, Devananda didn't like the idea.

And it definitely makes a quick start guide a bit harder to follow. I
already imagine how many people will forget about this pinning (either
to do it, or do update when they need new features).

The version depends on the environment you are running against - why not
treat it as such?

d.1) you start to break people \o/ that's not a theoretical concern:
our
downstream tooling did get broken by updating to newer ironicclient
from git

As I said before, we need to encourage folks to pin client versions if
they don't want to break. I'm probably alone here, but I would even
propose making the version required. Force people to think about what
they are doing. If folks are okay with being broken, they can pass
"latest".

Could be a good default for devstack btw

d.2) you require complex version negotiations on the client side.
Otherwise
imaging CLI tool defaulting to 1.6 will issue node-create to Ironic
supporting only 1.5. Guess what? It will fail despite node-create being
very
old feature. Again, that's not something theoretical: that's how we
broke
TripleO CI.

Again, pin it.

e) Every microversion should be fully tested separately. Which ended up
in
Ironic having 4 versions 1.2-1.5 that were never ever gate tested. Even
worse, initially, our gate tested only the oldest version 1.1, but we
solved
it (though it took time to realize). The only good thing here is that
these
versions 1.2-1.5 were probably never used by anyone.

Hi. I've used some of these. :)

You didn't tell me last time we talked :) note, that you didn't use
them, unless you explicitly requested, because IIRC we never defaulted
our client to any of these. So for most people, even deploying from
master, it was 1.1 -> 1.6.

// jim

[0]
https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/__init__.py#L59-63
[1]
https://review.openstack.org/#/c/188873/1/ironic/api/controllers/v1/node.py

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of
the latter at all). I'm very opposed to continuing doing it in Ironic,
and
I'm going to propose patch stopping gating Kilo changes (non-breaking
obviously).

Hope that helps,
Dmitry

>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks & Regards
Ghanshyam Mann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

On Thu, Jun 25, 2015 at 5:18 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com wrote:
Sorry for late response here,

2015-06-20 9:14 GMT+09:00 Devananda van der Veen devananda.vdv@gmail.com:

Long version...
Every HTTP response from Ironic today includes three headers: min, max, and
version. The service can present an older API version, as long as it is
greater-than-or-equal-to the minimum supported version, even if that version
is incompatible with the maximum supported version. It does this by
rewriting responses to match what was expected in the requested (older)
version.

When the newer version is identical for all interfaces present in the
older version, this can be called compatible. Dmitry's point is that we
don't need to hide newer interfaces from users who request an older API
version, because the client won't know or care about things that weren't in
the version it requested.

However, we do need to signal their presence, and we don't have a good
means for that right now. We also need to signal to the client that the
response given is "compatible with" a certain "age" of API, even if it's not
exactly the same. And we don't have any means for that, either.

Time for an example....

Let's say that an incompatible change was made in v1.3. Let's also say that
a change was made in v1.5 that added a new endpoint. Today, this is what the
response headers would look like when calling a server running v1.5.

a) client requests v1.2, receives headers (min: 1.1, max: 1.5, current: 1.2)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5, current 1.4)
c) client requests v1.5, receives headers (min: 1.1, max: 1.5, current: 1.5)

What we have implemented today is that in case (b), the service will hide
any changes that we introduced in v1.5. But those changes did not affect any
functionality of the v1.4 API, so Dmitry objects to this. So do I.

The issue at hand is the response in case (b) ... though after spending the
last several months working on api versioning, I actually think all of those
are poor responses.

What I believe we should have:
a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.1)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)
b) client requests v1.5, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)

Yes -- (b) and (c) are identical responses.

Discuss.

I think it is good that backwards compatible changes(new features) are
available on older microversion also only if the clouds which are
used by users continue upgrading.

I think Sophia's role on "The Backwards Compatibility Fallacy" of
Sean's blog[1] has answered to this question, but I'd like to try
explaining it here for considering Ironic situation.
As the example, there are multiple public clouds which provide
different max microversions like:

Cloud A: Max microversion: v1.5
Cloud B: Max microversion: v1.1

A user wrote his own application for running on cloud A and specifying
v1.1 on the first application implementation.
The first application used small number of APIs, and he wanted to
extend the application.
If all backwards compatible changes(v1.2 - v1.5) appear on lower
microversion(in this case v1.1), he can use all new features even if
specifying v1.1.
That seemed really great for users at this time, and he extended the
application for using all features as possible.
but the specified microversion still is v1.1 because his application
worked fine even if using newer features.

After that, he needed to switch to the other cloud because of cost
merit or something.
The specified microversion was v1.1, so he did think his application
can work fine on cloud B also because of cloud B's max microversion.
But yes, his application could not work because his application had
already used newer features which are implemented on v1.2+.
In the real world, there are a lot of clouds and it is easy to imagine
this situation.

This is really nice point. We should give power to users to decide whether they
need/want to use/get the new features introduced in new microversion
irrespective
of it is backward compatible or not.

Current microversion implementation of Nova is blocking this situation
by making backwards every compatible change appear on each
microversion.
Nova team needs to consider interoperability between clouds so well
because Nova API is one of general external interfaces for end users.

On the other hand, Ironic API is for administrators, not for end users.
I am imaging that:
* Some administrator wrote his application for using Ironic API.
* From the viewpoint of administrator, the switching destination cloud
in newer in most cases.
* The application can continue working on newer clouds even after
switching many times.

So I feel the above interoperable issue example would not happen on
Ironic in most cases unless hiding backwards compatible changes on
lower microversion.
I guess this is the difference between Nova and Ironic on
interoperability discussion.

I cannot/don't want to enforce Ironic way at all, and it's fine to
find the best way on each project as OSS projects.
But only my concern here is that we cannot use "Microversions" as a
perfect keyword for OpenStack interoperability on whole OpenStack
projects if Ironic goes to the other way.

Thanks
Ken Ohmichi


[1]: https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks & Regards
Ghanshyam Mann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

26 июня 2015 г. 2:47 пользователь "GHANSHYAM MANN" ghanshyammann@gmail.com
написал:

On Sat, Jun 20, 2015 at 9:14 AM, Devananda van der Veen
devananda.vdv@gmail.com wrote:

Almost all of our discussions so far on this topic have left something
out,
which Monty pointed out to me last week. I'm following up now because
E_TRAVEL...

tldr;
What we're versioning here are API's, not packages. It's not a question
of
numbering and dependency ordering, but of communicating support[ed|able]
interfaces between running services. Libtool is more relevant than
semver.

http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html

Right now we lack a means to express that the API response is
"compatible-with" a particular previously-released version of the API.
If we
had that, instead of "current-version", I believe we would have much
happier
users (and a happier Dmitry and jroll).

Long version...
Every HTTP response from Ironic today includes three headers: min, max,
and
version. The service can present an older API version, as long as it is
greater-than-or-equal-to the minimum supported version, even if that
version
is incompatible with the maximum supported version. It does this by
rewriting responses to match what was expected in the requested (older)
version.

When the newer version is identical for all interfaces present in the
older version, this can be called compatible. Dmitry's point is that we
don't need to hide newer interfaces from users who request an older API
version, because the client won't know or care about things that
weren't in
the version it requested.

However, we do need to signal their presence, and we don't have a good
means for that right now. We also need to signal to the client that the
response given is "compatible with" a certain "age" of API, even if
it's not
exactly the same. And we don't have any means for that, either.

Time for an example....

Let's say that an incompatible change was made in v1.3. Let's also say
that
a change was made in v1.5 that added a new endpoint. Today, this is
what the
response headers would look like when calling a server running v1.5.

a) client requests v1.2, receives headers (min: 1.1, max: 1.5, current:
1.2)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5, current
1.4)
c) client requests v1.5, receives headers (min: 1.1, max: 1.5, current:
1.5)

What we have implemented today is that in case (b), the service will
hide
any changes that we introduced in v1.5. But those changes did not
affect any
functionality of the v1.4 API, so Dmitry objects to this. So do I.

The issue at hand is the response in case (b) ... though after spending
the
last several months working on api versioning, I actually think all of
those
are poor responses.

What I believe we should have:
a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.1)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)
b) client requests v1.5, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)

This is nice idea to return "compatible-with" information. But I feel
each microversion either backward compatible or not should have their
unique output only for what they had been introduced (not include new
version changes).

Sigh. Please provide some justification.

If older version shows what is in newer version(backward comp) then,
there is no meaning
of introducing those compatible changes as microversion. And if so
then it blank out whole
idea of publishing changes/features through microversion even they are
backward compatible.

The main reason for versioning is feature discovery. It had nothing to do
with feature hiding. Please read the whole thread to see why feature hiding
is actually harmful.

How about combining those means older version do not include anything
from newer version
but it returns "compatible-with:" information also which can be useful
for user to bump version easily.

Yes -- (b) and (c) are identical responses.

Discuss.

-Devananda

On Tue, Jun 16, 2015 at 7:13 AM Dmitry Tantsur dtantsur@redhat.com
wrote:

On 06/16/2015 03:47 PM, Jim Rollenhagen wrote:

On Tue, Jun 16, 2015 at 08:56:37AM +0200, Dmitry Tantsur wrote:

On 06/04/2015 08:58 AM, Xu, Hejie wrote:

Hi, guys,
I’m working on adding Microversion into the API-WG’s guideline
which
make sure we have consistent Microversion behavior in the API for
user.
The Nova and Ironic already have Microversion implementation, and
as I
know Magnum _https://review.openstack.org/#/c/184975/_ is going to
implement Microversion also.
Hope all the projects which support( or plan to) Microversion can
join
the review of guideline.
The Mircoversion specification(this almost copy from nova-specs):
_https://review.openstack.org/#/c/187112_
And another guideline for when we should bump Mircoversion
_https://review.openstack.org/#/c/187896/_
As I know, there already have a little different between Nova and
Ironic’s implementation. Ironic return min/max version when the
requested
version doesn’t support in server by http-headers. There isn’t such
thing in nova. But that is something for version negotiation we
need
for
nova also.
Sean have pointed out we should use response body instead of http
headers, the body can includes error message. Really hope ironic
team
can take a
look at if you guys have compelling reason for using http headers.
And if we think return body instead of http headers, we probably
need
think about back-compatible also. Because Microversion itself isn’t
versioned.
So I think we should keep those header for a while, does make
sense?
Hope we have good guideline for Microversion, because we only can
change
Mircoversion itself by back-compatible way.
Thanks
Alex Xu

Hi all!

I'd like to try put in feedback based on living with microversions
in
Kilo
release of Ironic.

And here's my take, based on my experiences. Keep in mind I'm a core
reviewer, a developer, and an operator of Ironic.

Thanks Jim, much appreciated!

From an ops perspective, our team has built our fair share of
tooling
to
help us run Ironic. Some of it uses the REST API via python or
node.js,
and of course we all use the CLI client often.

We also continuously deploy Ironic, for full transparency. My
experience
is not with how this works every 6 months, but in the day-to-day.

First of all, after talking to folks off-list, I realized that we
all,
and
the spec itself, confuse 3 aspects of microversion usage:

  1. protecting from breaking changes.
    This is clearly a big win from user's point of view, and it allowed
    us
    to
    conduct painful change with renaming an important node state in our
    state
    machine. It will allows us even worse change this cycle: change of
    the
    default state.

+1. Good stuff. My tooling doesn't break when I upgrade. Yay.

  1. API discoverability.
    While I believe that there maybe be better implementation of this
    idea,
    I
    think I got it now. People want services to report API versions they
    support. People want to be able to request a specific version, and
    fail
    early if it is not present. Also +1 from me.

I don't tend to personally do this. I usually am aware of what
version
of Ironic I'm running against. However I see how this could be useful
for other folks.

I do, however, use the versions to say, "Oh, I can now request 1.5
which
has logical names! That's useful, let's set those to the names in our
CMDB." Now my tooling that interacts with the CMDB and Ironic can
look
at the version and decide to use node.name instead of the old hack we
used to use.

  1. hiding new features from older clients
    This is not directly stated by the spec, but many people imply it,
    and
    Nova
    and Ironic did it in Kilo. I want us to be clear: it is not the
    same as

2.

You can report versions, but still allow new features to be used.

This is still totally useful. If you know what version you are
running
against, you know exactly what features are available.

"You know" is about #2 - that's where confusion is :)
so if you know, that moving to inspection state is disallowed for your
tooling (but not for the whole system!), what does it give you?

I think the disconnect here is that we don't expect users (whether
those
are people or computers) to explicitly request a version. We need to
message better that if you are using Ironic or building a tool
against
Ironic's API, you should be pinning the version. We also need to take
this comment block[0] and put it in our docs, so users know what each
version does.

Knowing that I get feature X when I upgrade to version Y is useful.

It is this particular thing that gets -2 from me, after I've seen
how
it
worked in practice, and that's why.

First of all, I don't believe anyone needs it. Seriously, I can't
imagine a
user asking "please prevent me from using non-breaking changes". And
attempt
to implement it was IMO a big failure for the following reasons:

a) It's hard to do. Even we, the core team, got confused, and for
non-core
people it took several iteration to do right. It's a big load on
both
developers and reviewers.

I do agree with this. It's been painful. However, I think we're
mostly
past that pain at this point. Does this patch[1] look like developer
pain?

It's not that painful to write. Now. When we have 10-20 version, it
probably will :)
anyway, it's hard to explain newcomers how to do it, and it's hard to
review the result. we failed at it, e.g. with error codes.

b) It's incomplete (at least for Ironic). We have several
API-exposed
things
that are just impossible to hide. Good example are node states: if
node
is
in a new state, we can't but expose it to older tooling. Our
free-form
JSON
fields properties, instanceinfo, driverinfo and
driverinternalinfo
are
examples as well. It's useless to speak about API contract, while we
have
those.

I somewhat agree here.

With node states, there are cases where we were able to hide it
(NOSTATE -> AVAILABLE), and cases where we were not (adding
MANAGEABLE).
However, this list of states is (AIUI) not part of the API contract;
rather the verbs available to move between states are.

What's the point in contract, if there are things not covered by it
that
drastically change the system behavior?

As far as JSON fields, we've never had a contract around what keys
are
available. Only the semantics of working with those fields, and which
fields exist.

ditto as above: you can request new features by modifying driver_info.

c) It gives additional push back to making (required) breaking
changes.
We
already got suggestions to have ONE MORE feature gating for breaking
changes. Reason: people will need to increase microversions to get
features,
and your breaking change will prevent it.

This is just silly. If 1.10 breaks a user, and the user wants 1.11,
they'll need to fix that breakage.

++ but not everyone agreed on the summit, when I was talking about
ENROLL state

d) It requires a hard compromise on the CLI tool. You either
default it
to
1.0 forever, and force all the people to get used to figuring out
version
numbers and using ironic --ironic-api-version x.y every time
(terrible
user experience), or you default it to some known good version,
bumping
it
from time to time. This, in turn, has 2 more serious problems:

I disagree that pinning a version all the time is a terrible
experience.
We already require a number of options for authentication
(OSUSERNAME,
OS
PASSWORD, etc etc). How many folks do you think type these in
every
time? Solution is simple: add IRONICAPIVERSION to whatever exports
the
other environment variables.

It's not that bad, especially if devstack/tripleo will provide some
reasonable default for you.

I remember, however, Devananda didn't like the idea.

And it definitely makes a quick start guide a bit harder to follow. I
already imagine how many people will forget about this pinning (either
to do it, or do update when they need new features).

The version depends on the environment you are running against - why
not
treat it as such?

d.1) you start to break people \o/ that's not a theoretical concern:
our
downstream tooling did get broken by updating to newer ironicclient
from git

As I said before, we need to encourage folks to pin client versions
if
they don't want to break. I'm probably alone here, but I would even
propose making the version required. Force people to think about
what
they are doing. If folks are okay with being broken, they can pass
"latest".

Could be a good default for devstack btw

d.2) you require complex version negotiations on the client side.
Otherwise
imaging CLI tool defaulting to 1.6 will issue node-create to
Ironic
supporting only 1.5. Guess what? It will fail despite node-create
being
very
old feature. Again, that's not something theoretical: that's how we
broke
TripleO CI.

Again, pin it.

e) Every microversion should be fully tested separately. Which
ended up
in
Ironic having 4 versions 1.2-1.5 that were never ever gate tested.
Even
worse, initially, our gate tested only the oldest version 1.1, but
we
solved
it (though it took time to realize). The only good thing here is
that
these
versions 1.2-1.5 were probably never used by anyone.

Hi. I've used some of these. :)

You didn't tell me last time we talked :) note, that you didn't use
them, unless you explicitly requested, because IIRC we never defaulted
our client to any of these. So for most people, even deploying from
master, it was 1.1 -> 1.6.

// jim

[0]

https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/__init__.py#L59-63

[1]

https://review.openstack.org/#/c/188873/1/ironic/api/controllers/v1/node.py

To sum this long post up, I'm seeing that hiding new features based
on
microversions brings much more problems, than it solves (I'm not
aware
of
the latter at all). I'm very opposed to continuing doing it in
Ironic,
and
I'm going to propose patch stopping gating Kilo changes
(non-breaking
obviously).

Hope that helps,
Dmitry

>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks & Regards
Ghanshyam Mann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by divius.inside_at_gma (1,080 points)   1
0 votes

26 июня 2015 г. 2:48 пользователь "GHANSHYAM MANN" ghanshyammann@gmail.com
написал:

On Thu, Jun 25, 2015 at 5:18 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Sorry for late response here,

2015-06-20 9:14 GMT+09:00 Devananda van der Veen <
devananda.vdv@gmail.com>:

Long version...
Every HTTP response from Ironic today includes three headers: min,
max, and
version. The service can present an older API version, as long as it is
greater-than-or-equal-to the minimum supported version, even if that
version
is incompatible with the maximum supported version. It does this by
rewriting responses to match what was expected in the requested (older)
version.

When the newer version is identical for all interfaces present in the
older version, this can be called compatible. Dmitry's point is that we
don't need to hide newer interfaces from users who request an older API
version, because the client won't know or care about things that
weren't in
the version it requested.

However, we do need to signal their presence, and we don't have a
good
means for that right now. We also need to signal to the client that the
response given is "compatible with" a certain "age" of API, even if
it's not
exactly the same. And we don't have any means for that, either.

Time for an example....

Let's say that an incompatible change was made in v1.3. Let's also say
that
a change was made in v1.5 that added a new endpoint. Today, this is
what the
response headers would look like when calling a server running v1.5.

a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
current: 1.2)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5, current
1.4)
c) client requests v1.5, receives headers (min: 1.1, max: 1.5,
current: 1.5)

What we have implemented today is that in case (b), the service will
hide
any changes that we introduced in v1.5. But those changes did not
affect any
functionality of the v1.4 API, so Dmitry objects to this. So do I.

The issue at hand is the response in case (b) ... though after
spending the
last several months working on api versioning, I actually think all of
those
are poor responses.

What I believe we should have:
a) client requests v1.2, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.1)
b) client requests v1.4, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)
b) client requests v1.5, receives headers (min: 1.1, max: 1.5,
compatible-with: 1.3)

Yes -- (b) and (c) are identical responses.

Discuss.

I think it is good that backwards compatible changes(new features) are
available on older microversion also only if the clouds which are
used by users continue upgrading.

I think Sophia's role on "The Backwards Compatibility Fallacy" of
Sean's blog[1] has answered to this question, but I'd like to try
explaining it here for considering Ironic situation.
As the example, there are multiple public clouds which provide
different max microversions like:

Cloud A: Max microversion: v1.5
Cloud B: Max microversion: v1.1

A user wrote his own application for running on cloud A and specifying
v1.1 on the first application implementation.
The first application used small number of APIs, and he wanted to
extend the application.
If all backwards compatible changes(v1.2 - v1.5) appear on lower
microversion(in this case v1.1), he can use all new features even if
specifying v1.1.
That seemed really great for users at this time, and he extended the
application for using all features as possible.
but the specified microversion still is v1.1 because his application
worked fine even if using newer features.

After that, he needed to switch to the other cloud because of cost
merit or something.
The specified microversion was v1.1, so he did think his application
can work fine on cloud B also because of cloud B's max microversion.
But yes, his application could not work because his application had
already used newer features which are implemented on v1.2+.
In the real world, there are a lot of clouds and it is easy to imagine
this situation.

By the way, feature hiding did not change the situation: the application
using new features won't work on older cloud, period.

>

This is really nice point. We should give power to users to decide
whether they
need/want to use/get the new features introduced in new microversion
irrespective
of it is backward compatible or not.

I'm already tried of asking why, but I'll try again: why? Did you even read
this thread? It's not power, it's limitation, and a pretty nasty one.

Current microversion implementation of Nova is blocking this situation
by making backwards every compatible change appear on each
microversion.
Nova team needs to consider interoperability between clouds so well
because Nova API is one of general external interfaces for end users.

On the other hand, Ironic API is for administrators, not for end users.
I am imaging that:
* Some administrator wrote his application for using Ironic API.
* From the viewpoint of administrator, the switching destination cloud
in newer in most cases.
* The application can continue working on newer clouds even after
switching many times.

So I feel the above interoperable issue example would not happen on
Ironic in most cases unless hiding backwards compatible changes on
lower microversion.
I guess this is the difference between Nova and Ironic on
interoperability discussion.

I cannot/don't want to enforce Ironic way at all, and it's fine to
find the best way on each project as OSS projects.
But only my concern here is that we cannot use "Microversions" as a
perfect keyword for OpenStack interoperability on whole OpenStack
projects if Ironic goes to the other way.

Thanks
Ken Ohmichi


[1]: https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2/

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks & Regards
Ghanshyam Mann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by divius.inside_at_gma (1,080 points)   1
0 votes

On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:
On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch: https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

If we were only working on a single install of OpenStack in the world,
like EC2, github, meetup, or any other proprietary API service out
there, it would be a different story. We'd control time's arrow, and
never would have to worry about "going back in time". But we're not
working on that. We're working on software we hope will bloom 10,000+
clouds. At all scales. And the challenge to provide a consistent
programming experience for application writers that want to work against
any of those clouds means we needed to come up with some new models here.

If you have other solutions to the time's arrow issue, that would be
great to here. But it is the crux of why the model looks like it does.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 06/26/2015 01:14 PM, Sean Dague wrote:
On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:

On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch: https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I clearly agree that my solutions do not solve this situation. Neither
does yours. Now let us see:

A compatible change is getting added and is guarded behind version 1.5.
A new developer starts requiring this new version, because she needs
this new feature (that's your assumption).

Then she points her application at cloud A. And it explodes. But with
different error message and probably a bit earlier. But explodes.

Which, by the way, means we still must know precisely which API version
is served by all clouds we might apply our utility to.

So I agree, hitting API version error could make a person realize that a
change to the utility is no longer compatible with the current version
of API. So if this change wasn't intended - fine. If it was (which is
the most likely situation), it won't help you.

By the way, I've heard some people wanting to deprecate API version with
time. If we do so, it will open some new horizons for breakages.
If we don't, we'll soon end put with dozens of version to support and
test. How to solve it? (in Ironic IIRC we just only gate test one
version, namely Kilo aka 1.6, which is very, very bad IMO).

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

I think that versioning actually encourages this (wrong) assumption.
Because versions grow with time naturally, and we, the developers, like
new and shiny stuff so much :) see below for my alternative idea.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

Note that if you rely on version 1.4 with feature hiding, your
application will also break. I.e. I agree it's a valid situation to fix,
I just don't see feature hiding fixing all cases of this problem.
While an API exposing feature explicitly might be more handy, imagine e.g.

$ curl http://ironic.host/v1/features
['inspection', 'raid', ...]

Exposing interface like that would encourage people to think about their
application in terms of mandatory and optional features, not some numbers.

If we were only working on a single install of OpenStack in the world,
like EC2, github, meetup, or any other proprietary API service out
there, it would be a different story. We'd control time's arrow, and
never would have to worry about "going back in time". But we're not
working on that. We're working on software we hope will bloom 10,000+
clouds. At all scales. And the challenge to provide a consistent
programming experience for application writers that want to work against
any of those clouds means we needed to come up with some new models here.

If you have other solutions to the time's arrow issue, that would be
great to here. But it is the crux of why the model looks like it does.

I see, thanks for explanation.

-Sean


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Dmitry_Tantsur (18,080 points)   2 3 7
0 votes

On 06/26/2015 07:43 AM, Dmitry Tantsur wrote:
On 06/26/2015 01:14 PM, Sean Dague wrote:

On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:

On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch: https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I clearly agree that my solutions do not solve this situation. Neither
does yours. Now let us see:

A compatible change is getting added and is guarded behind version 1.5.
A new developer starts requiring this new version, because she needs
this new feature (that's your assumption).

Then she points her application at cloud A. And it explodes. But with
different error message and probably a bit earlier. But explodes.

Which, by the way, means we still must know precisely which API version
is served by all clouds we might apply our utility to.

But it fails with "Your application is attempting to use API 1.5 which
is not supported in this cloud". Because if you don't specify a version,
you get the base 1.0, and never had new features.

That's an error which is extremely clear to understand what went wrong.
Has docs to figure out if the application could work at an earlier
release version, and provides the ability for the client to do some
selection logic based on supported versions.

-Sean

So I agree, hitting API version error could make a person realize that a
change to the utility is no longer compatible with the current version
of API. So if this change wasn't intended - fine. If it was (which is
the most likely situation), it won't help you.

By the way, I've heard some people wanting to deprecate API version with
time. If we do so, it will open some new horizons for breakages.
If we don't, we'll soon end put with dozens of version to support and
test. How to solve it? (in Ironic IIRC we just only gate test one
version, namely Kilo aka 1.6, which is very, very bad IMO).

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

I think that versioning actually encourages this (wrong) assumption.
Because versions grow with time naturally, and we, the developers, like
new and shiny stuff so much :) see below for my alternative idea.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

Note that if you rely on version 1.4 with feature hiding, your
application will also break. I.e. I agree it's a valid situation to fix,
I just don't see feature hiding fixing all cases of this problem.
While an API exposing feature explicitly might be more handy, imagine e.g.

$ curl http://ironic.host/v1/features
['inspection', 'raid', ...]

Exposing interface like that would encourage people to think about their
application in terms of mandatory and optional features, not some numbers.

Yes, version number is being used as a proxy for a feature description.
Which is what version numbers are used for in software. It's a way to
let people think about bundles of things, instead of having to mentally
map out every single feature in their head.

Maybe we can some day to a place where we can have an API that's fully
feature describing, and doesn't need numbers. But after 2.5 years of
trying to do a major rewrite of the Nova API, it was pretty clear that
APIs of any reasonable size just can't do that. And can't rely on
software to keep up with your changes. This provides a mechanism for
evolution with a stability contract that API 1.4 is going to keep
looking exactly the same over time. So you can upgrade client code on
your schedule, not on the schedule of the server upgrades.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 06/26/2015 04:08 PM, Sean Dague wrote:
On 06/26/2015 07:43 AM, Dmitry Tantsur wrote:

On 06/26/2015 01:14 PM, Sean Dague wrote:

On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:

On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch: https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I clearly agree that my solutions do not solve this situation. Neither
does yours. Now let us see:

A compatible change is getting added and is guarded behind version 1.5.
A new developer starts requiring this new version, because she needs
this new feature (that's your assumption).

Then she points her application at cloud A. And it explodes. But with
different error message and probably a bit earlier. But explodes.

Which, by the way, means we still must know precisely which API version
is served by all clouds we might apply our utility to.

But it fails with "Your application is attempting to use API 1.5 which
is not supported in this cloud". Because if you don't specify a version,
you get the base 1.0, and never had new features.

That's an error which is extremely clear to understand what went wrong.
Has docs to figure out if the application could work at an earlier
release version, and provides the ability for the client to do some
selection logic based on supported versions.

I agree that error is clear. I'm not sure it's worth doing despite all
the problems that I mentioned in the beginning of this thread. In
particular, situation around official CLI and testing.

E.g. do you agree with Jim that we should make API version a required
argument for both CLI and Python library?

-Sean

So I agree, hitting API version error could make a person realize that a
change to the utility is no longer compatible with the current version
of API. So if this change wasn't intended - fine. If it was (which is
the most likely situation), it won't help you.

By the way, I've heard some people wanting to deprecate API version with
time. If we do so, it will open some new horizons for breakages.
If we don't, we'll soon end put with dozens of version to support and
test. How to solve it? (in Ironic IIRC we just only gate test one
version, namely Kilo aka 1.6, which is very, very bad IMO).

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

I think that versioning actually encourages this (wrong) assumption.
Because versions grow with time naturally, and we, the developers, like
new and shiny stuff so much :) see below for my alternative idea.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

Note that if you rely on version 1.4 with feature hiding, your
application will also break. I.e. I agree it's a valid situation to fix,
I just don't see feature hiding fixing all cases of this problem.
While an API exposing feature explicitly might be more handy, imagine e.g.

$ curl http://ironic.host/v1/features
['inspection', 'raid', ...]

Exposing interface like that would encourage people to think about their
application in terms of mandatory and optional features, not some numbers.

Yes, version number is being used as a proxy for a feature description.
Which is what version numbers are used for in software. It's a way to
let people think about bundles of things, instead of having to mentally
map out every single feature in their head.

Why do you think it's good? People are using features, not numbers. They
don't need to map out features - on the contrary they do need to map
versions to features. We've reached 1.8 with Ironic - and I (the core)
don't remember what were these version about, except for 1.6.

Maybe we can some day to a place where we can have an API that's fully
feature describing, and doesn't need numbers. But after 2.5 years of
trying to do a major rewrite of the Nova API, it was pretty clear that
APIs of any reasonable size just can't do that. And can't rely on
software to keep up with your changes. This provides a mechanism for
evolution with a stability contract that API 1.4 is going to keep
looking exactly the same over time. So you can upgrade client code on
your schedule, not on the schedule of the server upgrades.

I don't see any problem with creating an API serving list of strings,
and then making contributors update this list every time they change
API. It's the same as making contributors bump version number.

-Sean


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Dmitry_Tantsur (18,080 points)   2 3 7
0 votes

On Fri, Jun 26, 2015 at 7:39 AM, Dmitry Tantsur dtantsur@redhat.com wrote:

On 06/26/2015 04:08 PM, Sean Dague wrote:

On 06/26/2015 07:43 AM, Dmitry Tantsur wrote:

On 06/26/2015 01:14 PM, Sean Dague wrote:

On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:

On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based on
microversions brings much more problems, than it solves (I'm not aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch: https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I clearly agree that my solutions do not solve this situation. Neither
does yours. Now let us see:

A compatible change is getting added and is guarded behind version 1.5.
A new developer starts requiring this new version, because she needs
this new feature (that's your assumption).

Then she points her application at cloud A. And it explodes. But with
different error message and probably a bit earlier. But explodes.

Which, by the way, means we still must know precisely which API version
is served by all clouds we might apply our utility to.

But it fails with "Your application is attempting to use API 1.5 which
is not supported in this cloud". Because if you don't specify a version,
you get the base 1.0, and never had new features.

That's an error which is extremely clear to understand what went wrong.
Has docs to figure out if the application could work at an earlier
release version, and provides the ability for the client to do some
selection logic based on supported versions.

I agree that error is clear. I'm not sure it's worth doing despite all the
problems that I mentioned in the beginning of this thread. In particular,
situation around official CLI and testing.

E.g. do you agree with Jim that we should make API version a required
argument for both CLI and Python library?

    -Sean

So I agree, hitting API version error could make a person realize that a
change to the utility is no longer compatible with the current version
of API. So if this change wasn't intended - fine. If it was (which is
the most likely situation), it won't help you.

By the way, I've heard some people wanting to deprecate API version with
time. If we do so, it will open some new horizons for breakages.
If we don't, we'll soon end put with dozens of version to support and
test. How to solve it? (in Ironic IIRC we just only gate test one
version, namely Kilo aka 1.6, which is very, very bad IMO).

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

I think that versioning actually encourages this (wrong) assumption.
Because versions grow with time naturally, and we, the developers, like
new and shiny stuff so much :) see below for my alternative idea.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

Note that if you rely on version 1.4 with feature hiding, your
application will also break. I.e. I agree it's a valid situation to fix,
I just don't see feature hiding fixing all cases of this problem.
While an API exposing feature explicitly might be more handy, imagine
e.g.

$ curl http://ironic.host/v1/features
['inspection', 'raid', ...]

Exposing interface like that would encourage people to think about their
application in terms of mandatory and optional features, not some
numbers.

Yes, version number is being used as a proxy for a feature description.
Which is what version numbers are used for in software. It's a way to
let people think about bundles of things, instead of having to mentally
map out every single feature in their head.

Why do you think it's good? People are using features, not numbers. They
don't need to map out features - on the contrary they do need to map
versions to features. We've reached 1.8 with Ironic - and I (the core)
don't remember what were these version about, except for 1.6.

Does Ironic have a document explaining each API version? I quick search in
the source tree didn't turn anything up. Being able to easily figure out
what each API version do is important for end users. As a user, when I see
the cloud I use now supports nova API 2.26 I want an easy way of figuring
out what that actually means.

To address this, nova has the following document:

http://docs.openstack.org/developer/nova/api_microversion_history.html

Maybe we can some day to a place where we can have an API that's fully
feature describing, and doesn't need numbers. But after 2.5 years of
trying to do a major rewrite of the Nova API, it was pretty clear that
APIs of any reasonable size just can't do that. And can't rely on
software to keep up with your changes. This provides a mechanism for
evolution with a stability contract that API 1.4 is going to keep
looking exactly the same over time. So you can upgrade client code on
your schedule, not on the schedule of the server upgrades.

I don't see any problem with creating an API serving list of strings, and
then making contributors update this list every time they change API. It's
the same as making contributors bump version number.

Nova has already been down that path, our microversion spec discusses a few
of the issues we found (It doesn't go into a lot of detail though):
http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html#problem-description

Additionally the extensions model doesn't support backwards incompatible
changes.

    -Sean


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Joe_Gordon (24,620 points)   2 5 8
0 votes

Dmitry,

Feature strings won't work. It only makes things more complex, without
addressing the underlying issue.

What happens when we change, even in a small way, the API surface exposed
by an existing feature? How is that change exposed -- unless we include a
version number along with your feature string? And now we're right back
where we started.

-Deva

On Fri, Jun 26, 2015, 07:41 Dmitry Tantsur dtantsur@redhat.com wrote:

On 06/26/2015 04:08 PM, Sean Dague wrote:

On 06/26/2015 07:43 AM, Dmitry Tantsur wrote:

On 06/26/2015 01:14 PM, Sean Dague wrote:

On 06/16/2015 09:51 AM, Dmitry Tantsur wrote:

On 06/16/2015 08:56 AM, Dmitry Tantsur wrote:

To sum this long post up, I'm seeing that hiding new features based
on
microversions brings much more problems, than it solves (I'm not
aware
of the latter at all). I'm very opposed to continuing doing it in
Ironic, and I'm going to propose patch stopping gating Kilo changes
(non-breaking obviously).

I'm talking about this patch:
https://review.openstack.org/#/c/192196/
We have to do it right now, as otherwise we can't test inspection in
tempest (it does not seem to be microversion-aware).

Dmitry,

How do you solve for the following situation?

2 Clouds (A and B), both being Continuously Deployed.

Assume both clouds start at same revision of code. At point in time T a
new "compatable change" is added to the API. For instance, another
field
returned by some resource.

Cloud B upgrades to that change.

Brand new developer shows up. Starts writing application against Cloud
B. Sees that change is available at version 1.4. Hard codes her
application to use this parameter.

Then she points her application at Cloud A. And it explodes.

I clearly agree that my solutions do not solve this situation. Neither
does yours. Now let us see:

A compatible change is getting added and is guarded behind version 1.5.
A new developer starts requiring this new version, because she needs
this new feature (that's your assumption).

Then she points her application at cloud A. And it explodes. But with
different error message and probably a bit earlier. But explodes.

Which, by the way, means we still must know precisely which API version
is served by all clouds we might apply our utility to.

But it fails with "Your application is attempting to use API 1.5 which
is not supported in this cloud". Because if you don't specify a version,
you get the base 1.0, and never had new features.

That's an error which is extremely clear to understand what went wrong.
Has docs to figure out if the application could work at an earlier
release version, and provides the ability for the client to do some
selection logic based on supported versions.

I agree that error is clear. I'm not sure it's worth doing despite all
the problems that I mentioned in the beginning of this thread. In
particular, situation around official CLI and testing.

E.g. do you agree with Jim that we should make API version a required
argument for both CLI and Python library?

  -Sean

So I agree, hitting API version error could make a person realize that a
change to the utility is no longer compatible with the current version
of API. So if this change wasn't intended - fine. If it was (which is
the most likely situation), it won't help you.

By the way, I've heard some people wanting to deprecate API version with
time. If we do so, it will open some new horizons for breakages.
If we don't, we'll soon end put with dozens of version to support and
test. How to solve it? (in Ironic IIRC we just only gate test one
version, namely Kilo aka 1.6, which is very, very bad IMO).

I feel like in your concerns there has been an assumption that the
operation across all clouds effectively always goes forwards. But
because we're trying to encourage and create a multicloud ecosystem a
user application might experience the world the following way.

Cloud A -> Cloud A' -> Cloud A'' -> Cloud D -> Cloud B' -> Cloud C ->
Cloud C'.

I think that versioning actually encourages this (wrong) assumption.
Because versions grow with time naturally, and we, the developers, like
new and shiny stuff so much :) see below for my alternative idea.

While no individual cloud is likely to downgrade code (though we can't
fully rule that out), the fact that we'd like applications to work
against a wide range of deployments means effectively applications are
going to experience the world as if the code bases both upgrade and
downgrade over time.

Which means that a change is only "compatable" if the inverse of the
change is also "compatable". So a field add is only compatable if the
field delete is also considered compatible, because people are going to
experience that when they hop to another cloud.

Which is also why feature hiding is a thing. Because we don't control
when every cloud is going to upgrade, and what they'll upgrade to. So
the best idea so far about getting this right is that API 1.4 is
exactly a specific surface. Features added in 1.5 are not visibile if
you ask for 1.4. Because if they were, and you now wrote applications
that used that bleed through, when you jump to a cloud without 1.5 yet
deployed, all your application code breaks.

Note that if you rely on version 1.4 with feature hiding, your
application will also break. I.e. I agree it's a valid situation to fix,
I just don't see feature hiding fixing all cases of this problem.
While an API exposing feature explicitly might be more handy, imagine
e.g.

$ curl http://ironic.host/v1/features
['inspection', 'raid', ...]

Exposing interface like that would encourage people to think about their
application in terms of mandatory and optional features, not some
numbers.

Yes, version number is being used as a proxy for a feature description.
Which is what version numbers are used for in software. It's a way to
let people think about bundles of things, instead of having to mentally
map out every single feature in their head.

Why do you think it's good? People are using features, not numbers. They
don't need to map out features - on the contrary they do need to map
versions to features. We've reached 1.8 with Ironic - and I (the core)
don't remember what were these version about, except for 1.6.

Maybe we can some day to a place where we can have an API that's fully
feature describing, and doesn't need numbers. But after 2.5 years of
trying to do a major rewrite of the Nova API, it was pretty clear that
APIs of any reasonable size just can't do that. And can't rely on
software to keep up with your changes. This provides a mechanism for
evolution with a stability contract that API 1.4 is going to keep
looking exactly the same over time. So you can upgrade client code on
your schedule, not on the schedule of the server upgrades.

I don't see any problem with creating an API serving list of strings,
and then making contributors update this list every time they change
API. It's the same as making contributors bump version number.

  -Sean


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2015 by Devananda_van_der_Ve (10,380 points)   2 3 5
...