settingsLogin | Registersettings

[OpenStack-DefCore] Huawei Juno based distro passes 2015.05; same code fails 2015.07. Incompatible API test change from 2015.05 to 2015.05/07

0 votes

Hi, folks.

 

This is the email I sent to the interop@openstack.orb and the co-chairs of DefCore.  From today's DefCore meeting, I am posting this to the mailing list for discussion and hopefully resolution.  Mark Voelker said that other vendors had also had the same issue.  TL;DR  a set of Nova tests changed between the time 2015.05 build SHA was captured and the 2015.07 SHA was captured.  The changed tests were intended for the LIberty release, as they were not proposed/merged until after the Kilo release  Kilo release was April; changes to tests were merged June 19 (definitely in the Liberty cycle).

 

Thanks,

--Rocky

 

Below, is more info on Huawei's situation:

 

 

Huawei has a problem we hope you can help us with.  Our FusionSphere 6.0, based on Juno, qualified for  the "interop" OpenStack Powered (tm) program on the 2015.05 DefCore tests.  We are now trying to qualify our Public cloud offering, running the exact same FusionSphere/OpenStack software but cannot pass a subset of the Nova tests.

 

Huawei uses Nova api V2 (not V2.1).  As such, our implementation has "Additional Properties" enabled and used.  These are supposedly valid operations for the Juno release.

 

The tests in question were changed via https://review.openstack.org/#/c/156130.  AdditionalProperties were changed to "False" in all of the tests included in the review.

 

The problem is that Juno was released in October of 2014, with AdditionalProperties (extenstions) allowed in the v2 apis.  The tests were changed for the Kilo release cycle, but affect the Juno releases under the Interop/DefCore/Refstack test selection process because of the nonbranching nature of Tempest. 

 

We think a waiver is appropriate in this situation, as the Huawei code has not changed, yet it now fails API tests that were valid three months earlier.  We think that including the v2 api in the changes in theseDefCore  tests was a mistake, as the discussion below indicates, and that more care needs to be exercised when test changes are made to enforce new behaviors for later release cycles.  We would be happy to demonstrate that our release still passes the failed tests under the older version of those tests that exist in 2015.05. 

 

We have been working diligently to get the certification on our Public Cloud product, and had hoped to qualify/certify before the release of the 2016.01 standards, as our development resources are currently focused on moving our product to a more current  OpenStack release.

 

Please advise us as to how we proceed from here.

 

 

Respectfully,

Rocky Grober and our FusionSphere team

 

an excerpt from one of the changed files:

 

The problem of a backward incompatibility to an API being introduced by test changes was mentioned in one an exchange in comments:

 

afazekas

Mar 17, 2015

Patch Set 31: Code-Review-1

Does not makes sense to prevent additional properties in tempest on something which allows extensions.

Ghanshyam Mann

Mar 18, 2015

Patch Set 31:

@afazekas - Thanks for review.

Actually all schema has been modified to work for all extension (as current tests runs for all extension enable).

As v2 (v2.1) API are frozen, tempest should have additionalProperties False to capture any unwanted changes in those API.

afazekas

Mar 18, 2015

Patch Set 31:

https://wiki.openstack.org/wiki/APIChangeGuidelines

"Adding a property to a resource representation" is generally acceptable.

"Changing or removing a property in a resource representation" OR "Changing the semantics of a property in a resource representation which may be supplied by clients" is generally not acceptable.

We do not need to prevent additional properties.

Ghanshyam Mann

Mar 19, 2015

Patch Set 31:

Yea, those are acceptable and it used to done by adding new extension etc. But as v2 nd v2.1 are frozen and no new extension for thosse. All new attributes etc can be added with microversion only. As it is done for first microversion - https://review.openstack.org/#/c/140313/

Please do let me know if m missing anything on this.

Ken'ichi Ohmichi

Mar 19, 2015

Patch Set 31: Code-Review+2

Yeah, Ghanshyam is right.

As http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html , new attributes will be added with microversions and v2(v2.1) should be frozen. So this change is necessary for blocking additional attributes on v2 and v2.1 API.

 


asked Jan 7, 2016 in defcore-committee by Rochelle_Grober (7,040 points)   1 3 3

3 Responses

0 votes

On Jan 6, 2016, at 8:55 PM, Rochelle Grober rochelle.grober@huawei.com wrote:

Hi, folks.

This is the email I sent to the interop@openstack.orb and the co-chairs of DefCore. From today's DefCore meeting, I am posting this to the mailing list for discussion and hopefully resolution. Mark Voelker said that other vendors had also had the same issue.

Thanks for writing in, Rocky. Actually what I said was that this potential issue had been discussed before, but that in 6.5 months this is the first time we’ve actually had someone report it. =) Here are the historical references I provided on the etherpad you originally posted this to yesterday for discussion:

===

I brought this up on the DefCore mailing list in June:

http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.html

I believe we discussed it some at the midcycle as well after having it on the weekly agenda for a couple of weeks:

https://etherpad.openstack.org/p/DefCoreFlag.5
https://etherpad.openstack.org/p/DefCoreFlag.6
https://etherpad.openstack.org/p/DefCoreFlag.MidCycle

And I believe it’s come up in-channel a couple of times too…one I could put my finger on quickly is here; there are probably others:

http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-defcore.2015-06-24.log.html#t2015-06-24T23:16:06

This originated with an announcement about the change on the openstack-dev list in February:

I’m not sure there was ever a formal statement about it, but I believe generally the consensus from nova-core et al was that vendor modification of API responses (even if they were adding additional info rather than changing or removing info) was never really ok from an interoperability standpoint anyway (and violated the API Change Guidelines per openstack-dev discussion above), and the tempest change was deemed to help make that more clear. There was some discussion [from John Garbutt and Ken’ichi Ohmichi whom I’ve CC'd here] around whether the change should apply to the 2.1 API only (instead of 2.0 and 2.1), but AFAIK the sentiment never really generated enough support for someone to bother submitting a patch to change it in the gate.

On the DefCore side, I recall some discussion that if there were any affected vendors (in 6.5 months this is actually the first time we’ve actually had someone report it) could simply use a version of Tempest that predates the change since we do not currently require a specific version of Tempest (though refstack-client does have a setting that uses a specific Tempest version specified by a SHA by default). I recall warning at the time that this probably wouldn’t work in the long run since bugfixes to Tempest would eventually mandate that a newer version be used (see IRC link above), but again: sentiment seemed to be that since modifying API responses was never going to be interoperable anyway, that might simply be both a stopgap folks could use in the short term and incentive to stop modifying API responses in the longer term.

===

So with that historical context provided: a couple of questions for you.

1.) Is the intent here to ask for a flag on the tests that are failing for you? If so, the most appropriate thing to do may be to open a request via gerrit and we’ll discuss it there.

2.) Also, are you attempting to pass 2015.05 or 2015.07? If the former, have you tried simply rolling back to an earlier Tempest SHA that predates the additionalProperties change as suggested above? I believe there’s a good chance you’d be able to pass 2015.05 by doing so. The additionalProperties change was f0c30bc, so 968f1b3 would be the commit before that.

3.) How are you injecting additional properties into the API responses? E.g. have you changed nova-api code to do so? I ask because the API code (other than extensions) is currently a designated section in both the 2015.05 and 2015.07 guidelines:

https://github.com/openstack/defcore/blob/master/2015.05.json#L540
https://github.com/openstack/defcore/blob/master/2015.07.json#L1468

At Your Service,

Mark T. Voelker

TL;DR a set of Nova tests changed between the time 2015.05 build SHA was captured and the 2015.07 SHA was captured. The changed tests were intended for the LIberty release, as they were not proposed/merged until after the Kilo release Kilo release was April; changes to tests were merged June 19 (definitely in the Liberty cycle).

Thanks,
--Rocky

Below, is more info on Huawei's situation:


Huawei has a problem we hope you can help us with. Our FusionSphere 6.0, based on Juno, qualified for the "interop" OpenStack Powered (tm) program on the 2015.05 DefCore tests. We are now trying to qualify our Public cloud offering, running the exact same FusionSphere/OpenStack software but cannot pass a subset of the Nova tests.

Huawei uses Nova api V2 (not V2.1). As such, our implementation has "Additional Properties" enabled and used. These are supposedly valid operations for the Juno release.

The tests in question were changed via https://review.openstack.org/#/c/156130. AdditionalProperties were changed to "False" in all of the tests included in the review.

The problem is that Juno was released in October of 2014, with AdditionalProperties (extenstions) allowed in the v2 apis. The tests were changed for the Kilo release cycle, but affect the Juno releases under the Interop/DefCore/Refstack test selection process because of the nonbranching nature of Tempest.

We think a waiver is appropriate in this situation, as the Huawei code has not changed, yet it now fails API tests that were valid three months earlier. We think that including the v2 api in the changes in theseDefCore tests was a mistake, as the discussion below indicates, and that more care needs to be exercised when test changes are made to enforce new behaviors for later release cycles. We would be happy to demonstrate that our release still passes the failed tests under the older version of those tests that exist in 2015.05.

We have been working diligently to get the certification on our Public Cloud product, and had hoped to qualify/certify before the release of the 2016.01 standards, as our development resources are currently focused on moving our product to a more current OpenStack release.

Please advise us as to how we proceed from here.


Respectfully,
Rocky Grober and our FusionSphere team

an excerpt from one of the changed files:
<image001.gif>

The problem of a backward incompatibility to an API being introduced by test changes was mentioned in one an exchange in comments:

afazekas
Mar 17, 2015

Patch Set 31: Code-Review-1

Does not makes sense to prevent additional properties in tempest on something which allows extensions.
Ghanshyam Mann
Mar 18, 2015

Patch Set 31:

@afazekas - Thanks for review.

Actually all schema has been modified to work for all extension (as current tests runs for all extension enable).

As v2 (v2.1) API are frozen, tempest should have additionalProperties False to capture any unwanted changes in those API.
afazekas
Mar 18, 2015

Patch Set 31:

https://wiki.openstack.org/wiki/APIChangeGuidelines

"Adding a property to a resource representation" is generally acceptable.

"Changing or removing a property in a resource representation" OR "Changing the semantics of a property in a resource representation which may be supplied by clients" is generally not acceptable.

We do not need to prevent additional properties.
Ghanshyam Mann
Mar 19, 2015

Patch Set 31:

Yea, those are acceptable and it used to done by adding new extension etc. But as v2 nd v2.1 are frozen and no new extension for thosse. All new attributes etc can be added with microversion only.
As it is done for first microversion -

https://review.openstack.org/#/c/140313/

Please do let me know if m missing anything on this.
Ken'ichi Ohmichi
Mar 19, 2015

Patch Set 31: Code-Review+2

Yeah, Ghanshyam is right.

As
http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
, new attributes will be added with microversions and v2(v2.1) should be frozen. So this change is necessary for blocking additional attributes
on v2 and v2.1 API.



Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
responded Jan 7, 2016 by Mark_Voelker (4,000 points)   2 3
0 votes

Hey folks,

We actually have been impacted by this exact same issue. We have been
working with our development teams to actually change some of the
parameters in our API so that it will not be an issue for us and in fact
we pass if we manually disable this change in Tempest (we did that
analysis back when the change first landed to isolate the difference in a
passing then suddenly failing run). I have mentioned in context of other
DefCore discussions a few times that this is an example of a Tempest
change, for something that is not technically covered by the DefCore
specification, that actually modifies the outcome of DefCore certification.

That having been said, this specific change is something we are behind
from an interop perspective, the question in my mind that remains is more
what the process for the future is about Tempest changes that impact the
DefCore Spec/Tests.

Thanks,

Sam

Sam Danes, Architect, Software Development - Test
sam.danes@rackspace.com
mobile: 412.689.1532

On 1/7/16, 9:14 AM, "Mark Voelker" mvoelker@vmware.com wrote:

On Jan 6, 2016, at 8:55 PM, Rochelle Grober
rochelle.grober@huawei.com wrote:

Hi, folks.

This is the email I sent to the interop@openstack.orb and the co-chairs
of DefCore. From today's DefCore meeting, I am posting this to the
mailing list for discussion and hopefully resolution. Mark Voelker said
that other vendors had also had the same issue.

Thanks for writing in, Rocky. Actually what I said was that this
potential issue had been discussed before, but that in 6.5 months this is
the first time we’ve actually had someone report it. =) Here are the
historical references I provided on the etherpad you originally posted
this to yesterday for discussion:

===

I brought this up on the DefCore mailing list in June:

http://lists.openstack.org/pipermail/defcore-committee/2015-June/000849.ht
ml

I believe we discussed it some at the midcycle as well after having it on
the weekly agenda for a couple of weeks:

https://etherpad.openstack.org/p/DefCoreFlag.5
https://etherpad.openstack.org/p/DefCoreFlag.6
https://etherpad.openstack.org/p/DefCoreFlag.MidCycle

And I believe it’s come up in-channel a couple of times too…one I could
put my finger on quickly is here; there are probably others:

http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-d
efcore.2015-06-24.log.html#t2015-06-24T23:16:06

This originated with an announcement about the change on the
openstack-dev list in February:

http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.ht
ml

I’m not sure there was ever a formal statement about it, but I believe
generally the consensus from nova-core et al was that vendor modification
of API responses (even if they were adding additional info rather than
changing or removing info) was never really ok from an interoperability
standpoint anyway (and violated the API Change Guidelines per
openstack-dev discussion above), and the tempest change was deemed to
help make that more clear. There was some discussion [from John Garbutt
and Ken’ichi Ohmichi whom I’ve CC'd here] around whether the change
should apply to the 2.1 API only (instead of 2.0 and 2.1), but AFAIK the
sentiment never really generated enough support for someone to bother
submitting a patch to change it in the gate.

On the DefCore side, I recall some discussion that if there were any
affected vendors (in 6.5 months this is actually the first time we’ve
actually had someone report it) could simply use a version of Tempest
that predates the change since we do not currently require a specific
version of Tempest (though refstack-client does have a setting that uses
a specific Tempest version specified by a SHA by default). I recall
warning at the time that this probably wouldn’t work in the long run
since bugfixes to Tempest would eventually mandate that a newer version
be used (see IRC link above), but again: sentiment seemed to be that
since modifying API responses was never going to be interoperable anyway,
that might simply be both a stopgap folks could use in the short term and
incentive to stop modifying API responses in the longer term.

===

So with that historical context provided: a couple of questions for you.

1.) Is the intent here to ask for a flag on the tests that are failing
for you? If so, the most appropriate thing to do may be to open a
request via gerrit and we’ll discuss it there.

2.) Also, are you attempting to pass 2015.05 or 2015.07? If the former,
have you tried simply rolling back to an earlier Tempest SHA that
predates the additionalProperties change as suggested above? I believe
there’s a good chance you’d be able to pass 2015.05 by doing so. The
additionalProperties change was f0c30bc, so 968f1b3 would be the commit
before that.

3.) How are you injecting additional properties into the API responses?
E.g. have you changed nova-api code to do so? I ask because the API code
(other than extensions) is currently a designated section in both the
2015.05 and 2015.07 guidelines:

https://github.com/openstack/defcore/blob/master/2015.05.json#L540
https://github.com/openstack/defcore/blob/master/2015.07.json#L1468

At Your Service,

Mark T. Voelker

TL;DR a set of Nova tests changed between the time 2015.05 build SHA
was captured and the 2015.07 SHA was captured. The changed tests were
intended for the LIberty release, as they were not proposed/merged until
after the Kilo release Kilo release was April; changes to tests were
merged June 19 (definitely in the Liberty cycle).

Thanks,
--Rocky

Below, is more info on Huawei's situation:


Huawei has a problem we hope you can help us with. Our FusionSphere
6.0, based on Juno, qualified for the "interop" OpenStack Powered (tm)
program on the 2015.05 DefCore tests. We are now trying to qualify our
Public cloud offering, running the exact same FusionSphere/OpenStack
software but cannot pass a subset of the Nova tests.

Huawei uses Nova api V2 (not V2.1). As such, our implementation has
"Additional Properties" enabled and used. These are supposedly valid
operations for the Juno release.

The tests in question were changed via
https://review.openstack.org/#/c/156130. AdditionalProperties were
changed to "False" in all of the tests included in the review.

The problem is that Juno was released in October of 2014, with
AdditionalProperties (extenstions) allowed in the v2 apis. The tests
were changed for the Kilo release cycle, but affect the Juno releases
under the Interop/DefCore/Refstack test selection process because of the
nonbranching nature of Tempest.

We think a waiver is appropriate in this situation, as the Huawei code
has not changed, yet it now fails API tests that were valid three months
earlier. We think that including the v2 api in the changes in
theseDefCore tests was a mistake, as the discussion below indicates,
and that more care needs to be exercised when test changes are made to
enforce new behaviors for later release cycles. We would be happy to
demonstrate that our release still passes the failed tests under the
older version of those tests that exist in 2015.05.

We have been working diligently to get the certification on our Public
Cloud product, and had hoped to qualify/certify before the release of
the 2016.01 standards, as our development resources are currently
focused on moving our product to a more current OpenStack release.

Please advise us as to how we proceed from here.


Respectfully,
Rocky Grober and our FusionSphere team

an excerpt from one of the changed files:
<image001.gif>

The problem of a backward incompatibility to an API being introduced by
test changes was mentioned in one an exchange in comments:

afazekas
Mar 17, 2015

Patch Set 31: Code-Review-1

Does not makes sense to prevent additional properties in tempest on
something which allows extensions.
Ghanshyam Mann
Mar 18, 2015

Patch Set 31:

@afazekas - Thanks for review.

Actually all schema has been modified to work for all extension (as
current tests runs for all extension enable).

As v2 (v2.1) API are frozen, tempest should have additionalProperties
False to capture any unwanted changes in those API.
afazekas
Mar 18, 2015

Patch Set 31:

https://wiki.openstack.org/wiki/APIChangeGuidelines

"Adding a property to a resource representation" is generally
acceptable.

"Changing or removing a property in a resource representation" OR
"Changing the semantics of a property in a resource representation which
may be supplied by clients" is generally not acceptable.

We do not need to prevent additional properties.
Ghanshyam Mann
Mar 19, 2015

Patch Set 31:

Yea, those are acceptable and it used to done by adding new extension
etc. But as v2 nd v2.1 are frozen and no new extension for thosse. All
new attributes etc can be added with microversion only.
As it is done for first microversion -

https://review.openstack.org/#/c/140313/

Please do let me know if m missing anything on this.
Ken'ichi Ohmichi
Mar 19, 2015

Patch Set 31: Code-Review+2

Yeah, Ghanshyam is right.

As

http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.h
tml
, new attributes will be added with microversions and v2(v2.1) should
be frozen. So this change is necessary for blocking additional attributes
on v2 and v2.1 API.



Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
responded Jan 7, 2016 by Sam_Danes (280 points)  
0 votes

Thanks, Sam!

Huawei's issue is that the same release passes 2015.05 and not 2015.07. This is exactly the problem Mark Voelker highlighted in the IRC log he included [0]. Reviewing Mark's references, a few questions/issues can be extracted.

  1. In the discussions, there are multiple times where Chris mentions that a vendor could just use an older version of the test. This raises the questions:

    • How would you do this with Refstack?
    • How would the Foundation know which version of the test was used if this could be done with Refstack?
    • How should test changes that change the effective API definition (as opposed to the documented definition) be handled?
  2. Refstack used labels for the 2015.04 and 2015.05 guidelines to fix the sha of the tests that were used to validate against a cloud, but switched to the idempotent tags for 2015.07. With the label, you get tests that are fixed in time, but with idempotent tags, the test could change and the tag would stay the same, as long as the test "container" was the same. So, even as we speak, 2015.07 and 2016.01 could be changing in ways that would cause a cloud that passed today to fail tomorrow. Is this statement true? If so, how do we deal with this?
  3. We need to be much clearer and more careful if we do not use point-in-time versions of tests to ensure that Clouds that have passed a guideline continue to pass that guideline as long as the OpenStack software has not been changed.
  4. We need to be much clearer and more careful about how undocumented aspects of the API should not be retroactively be considered "incorrect" unless documented as such.

It turns out that Huawei still has time to qualify its public cloud offering under the 2015.05 guidelines, which use the branch tag, if they can get the tests filed before the board meeting. But, I want to stress that they want to test against the latest guidelines approved, yet because of the test change, their Juno distro would need to use some tests that are earlier versions of the current tests to pass.

Huawei tries to stay on the most current guidelines for the release have in their products and will continue to do so. We just need to have tests stable enough to not disrupt products we have in maintenance mode.

--Rocky

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-defcore.2015-06-24.log.html#t2015-06-24T23:16:06

-----Original Message-----
From: Sam Danes [mailto:sam.danes@RACKSPACE.COM]
Sent: Thursday, January 07, 2016 10:53 AM
To: Mark Voelker; Rochelle Grober
Cc: Chenrui (A); Liyongle (Fred); defcore-
committee@lists.openstack.org; Rick Lopez
Subject: Re: [OpenStack-DefCore] Huawei Juno based distro passes
2015.05; same code fails 2015.07. Incompatible API test change from
2015.05 to 2015.05/07

Hey folks,

We actually have been impacted by this exact same issue. We have been
working with our development teams to actually change some of the
parameters in our API so that it will not be an issue for us and in
fact
we pass if we manually disable this change in Tempest (we did that
analysis back when the change first landed to isolate the difference in
a
passing then suddenly failing run). I have mentioned in context of
other
DefCore discussions a few times that this is an example of a Tempest
change, for something that is not technically covered by the DefCore
specification, that actually modifies the outcome of DefCore
certification.

That having been said, this specific change is something we are behind
from an interop perspective, the question in my mind that remains is
more
what the process for the future is about Tempest changes that impact
the
DefCore Spec/Tests.

Thanks,

Sam

Sam Danes, Architect, Software Development - Test
sam.danes@rackspace.com
mobile: 412.689.1532
https://www.rackspacemarketing.com/signatyourEmail/
https://www.linkedin.com/in/samueldanes

On 1/7/16, 9:14 AM, "Mark Voelker" mvoelker@vmware.com wrote:

On Jan 6, 2016, at 8:55 PM, Rochelle Grober
rochelle.grober@huawei.com wrote:

Hi, folks.

This is the email I sent to the interop@openstack.orb and the co-
chairs
of DefCore. From today's DefCore meeting, I am posting this to the
mailing list for discussion and hopefully resolution. Mark Voelker
said
that other vendors had also had the same issue.

Thanks for writing in, Rocky. Actually what I said was that this
potential issue had been discussed before, but that in 6.5 months this
is
the first time we’ve actually had someone report it. =) Here are the
historical references I provided on the etherpad you originally posted
this to yesterday for discussion:

===

I brought this up on the DefCore mailing list in June:

http://lists.openstack.org/pipermail/defcore-committee/2015-
June/000849.ht
ml

I believe we discussed it some at the midcycle as well after having it
on
the weekly agenda for a couple of weeks:

https://etherpad.openstack.org/p/DefCoreFlag.5
https://etherpad.openstack.org/p/DefCoreFlag.6
https://etherpad.openstack.org/p/DefCoreFlag.MidCycle

And I believe it’s come up in-channel a couple of times too…one I
could
put my finger on quickly is here; there are probably others:

http://eavesdrop.openstack.org/irclogs/%23openstack-
defcore/%23openstack-d
efcore.2015-06-24.log.html#t2015-06-24T23:16:06

This originated with an announcement about the change on the
openstack-dev list in February:

http://lists.openstack.org/pipermail/openstack-dev/2015-
February/057613.ht
ml

I’m not sure there was ever a formal statement about it, but I believe
generally the consensus from nova-core et al was that vendor
modification
of API responses (even if they were adding additional info rather than
changing or removing info) was never really ok from an
interoperability
standpoint anyway (and violated the API Change Guidelines per
openstack-dev discussion above), and the tempest change was deemed to
help make that more clear. There was some discussion [from John
Garbutt
and Ken’ichi Ohmichi whom I’ve CC'd here] around whether the change
should apply to the 2.1 API only (instead of 2.0 and 2.1), but AFAIK
the
sentiment never really generated enough support for someone to bother
submitting a patch to change it in the gate.

On the DefCore side, I recall some discussion that if there were any
affected vendors (in 6.5 months this is actually the first time we’ve
actually had someone report it) could simply use a version of Tempest
that predates the change since we do not currently require a specific
version of Tempest (though refstack-client does have a setting that
uses
a specific Tempest version specified by a SHA by default). I recall
warning at the time that this probably wouldn’t work in the long run
since bugfixes to Tempest would eventually mandate that a newer
version
be used (see IRC link above), but again: sentiment seemed to be that
since modifying API responses was never going to be interoperable
anyway,
that might simply be both a stopgap folks could use in the short term
and
incentive to stop modifying API responses in the longer term.

===

So with that historical context provided: a couple of questions for
you.

1.) Is the intent here to ask for a flag on the tests that are
failing
for you? If so, the most appropriate thing to do may be to open a
request via gerrit and we’ll discuss it there.

2.) Also, are you attempting to pass 2015.05 or 2015.07? If the
former,
have you tried simply rolling back to an earlier Tempest SHA that
predates the additionalProperties change as suggested above? I
believe
there’s a good chance you’d be able to pass 2015.05 by doing so. The
additionalProperties change was f0c30bc, so 968f1b3 would be the
commit
before that.

3.) How are you injecting additional properties into the API
responses?
E.g. have you changed nova-api code to do so? I ask because the API
code
(other than extensions) is currently a designated section in both the
2015.05 and 2015.07 guidelines:

https://github.com/openstack/defcore/blob/master/2015.05.json#L540
https://github.com/openstack/defcore/blob/master/2015.07.json#L1468

At Your Service,

Mark T. Voelker

TL;DR a set of Nova tests changed between the time 2015.05 build
SHA
was captured and the 2015.07 SHA was captured. The changed tests
were
intended for the LIberty release, as they were not proposed/merged
until
after the Kilo release Kilo release was April; changes to tests were
merged June 19 (definitely in the Liberty cycle).

Thanks,
--Rocky

Below, is more info on Huawei's situation:

Huawei has a problem we hope you can help us with. Our FusionSphere
6.0, based on Juno, qualified for the "interop" OpenStack Powered
(tm)
program on the 2015.05 DefCore tests. We are now trying to qualify
our
Public cloud offering, running the exact same FusionSphere/OpenStack
software but cannot pass a subset of the Nova tests.

Huawei uses Nova api V2 (not V2.1). As such, our implementation has
"Additional Properties" enabled and used. These are supposedly valid
operations for the Juno release.

The tests in question were changed via
https://review.openstack.org/#/c/156130. AdditionalProperties were
changed to "False" in all of the tests included in the review.

The problem is that Juno was released in October of 2014, with
AdditionalProperties (extenstions) allowed in the v2 apis. The tests
were changed for the Kilo release cycle, but affect the Juno releases
under the Interop/DefCore/Refstack test selection process because of
the
nonbranching nature of Tempest.

We think a waiver is appropriate in this situation, as the Huawei
code
has not changed, yet it now fails API tests that were valid three
months
earlier. We think that including the v2 api in the changes in
theseDefCore tests was a mistake, as the discussion below indicates,
and that more care needs to be exercised when test changes are made
to
enforce new behaviors for later release cycles. We would be happy to
demonstrate that our release still passes the failed tests under the
older version of those tests that exist in 2015.05.

We have been working diligently to get the certification on our
Public
Cloud product, and had hoped to qualify/certify before the release of
the 2016.01 standards, as our development resources are currently
focused on moving our product to a more current OpenStack release.

Please advise us as to how we proceed from here.

Respectfully,
Rocky Grober and our FusionSphere team

an excerpt from one of the changed files:
<image001.gif>

The problem of a backward incompatibility to an API being introduced
by
test changes was mentioned in one an exchange in comments:

afazekas
Mar 17, 2015

Patch Set 31: Code-Review-1

Does not makes sense to prevent additional properties in tempest on
something which allows extensions.
Ghanshyam Mann
Mar 18, 2015

Patch Set 31:

@afazekas - Thanks for review.

Actually all schema has been modified to work for all extension (as
current tests runs for all extension enable).

As v2 (v2.1) API are frozen, tempest should have
additionalProperties
False to capture any unwanted changes in those API.
afazekas
Mar 18, 2015

Patch Set 31:

https://wiki.openstack.org/wiki/APIChangeGuidelines

"Adding a property to a resource representation" is generally
acceptable.

"Changing or removing a property in a resource representation" OR
"Changing the semantics of a property in a resource representation
which
may be supplied by clients" is generally not acceptable.

We do not need to prevent additional properties.
Ghanshyam Mann
Mar 19, 2015

Patch Set 31:

Yea, those are acceptable and it used to done by adding new
extension
etc. But as v2 nd v2.1 are frozen and no new extension for thosse.
All
new attributes etc can be added with microversion only.
As it is done for first microversion -

https://review.openstack.org/#/c/140313/

Please do let me know if m missing anything on this.
Ken'ichi Ohmichi
Mar 19, 2015

Patch Set 31: Code-Review+2

Yeah, Ghanshyam is right.

As

http://lists.openstack.org/pipermail/openstack-dev/2015-
February/057613.h
tml
, new attributes will be added with microversions and v2(v2.1)
should
be frozen. So this change is necessary for blocking additional
attributes
on v2 and v2.1 API.


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-
committee


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee


Defcore-committee mailing list
Defcore-committee@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
responded Jan 9, 2016 by Rochelle_Grober (7,040 points)   1 3 3
...