settingsLogin | Registersettings

[openstack-dev] [qa] [patrole] Question regarding patrole release management

0 votes

Hi all,

I have a question regarding patrole and release management. Many projects like heat or murano have a tempest plugin within their repos, so by extension their tempest plugins have releases, as the projects change over time. However, since patrole is just a tempest plugin, yet heavily reliant on tempest, how should patrole do release management? Am I correct in thinking that it should, in the first place? With nova-network and other APIs slated for deprecation in Pike and beyond, Patrole will logically have to continuously be maintained to keep up, meaning that older tests, just like with Tempest, will have to be phased out. If Patrole, then, does not have releases, then older release-dependent tests and functionality will over time be lost.

Thank You,
Felipe Monteiro


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 21, 2017 in openstack-dev by MONTEIRO,_FELIPE_C (520 points)  

3 Responses

0 votes

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C fm577c@att.com wrote:
Hi all,

I have a question regarding patrole and release management. Many projects
like heat or murano have a tempest plugin within their repos, so by
extension their tempest plugins have releases, as the projects change over
time. However, since patrole is just a tempest plugin, yet heavily reliant
on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

Am I correct in
thinking that it should, in the first place? With nova-network and other
APIs slated for deprecation in Pike and beyond, Patrole will logically have
to continuously be maintained to keep up, meaning that older tests, just
like with Tempest, will have to be phased out. If Patrole, then, does not
have releases, then older release-dependent tests and functionality will
over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.

Thank You,

Felipe Monteiro

..1 https://wiki.openstack.org/wiki/QA/releases
..2 https://review.openstack.org/#/c/391113/13

-gmann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 19, 2017 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

Thanks Ghanshyam,

That was really helpful. If Tempest is versioned via tags, then I agree that Patrole should do the same thing. I didn't quite know what you meant by "feature flag" but after finding [0] it became apparent to me: Patrole has been doing this wherever Tempest does it. Granted, we probably need more participation in the project to catch minute details like that, as we currently have a large body of tests, but currently we have been using feature flags where appropriate.

Case 3 as you mentioned is something that we've already had to account for: When Nova and then Keystone moved their policies into code, we had to add logic to the framework to account for those changes. Case 3 in theory would not be a problem, if there was a way to verify whether a policy action was enforced, other than by checking logs. If there was an API in OpenStack that could be called to retrieve oslo_policy enforcements per request id, then Case 3 would just result in an error being thrown by Patrole for a given test, which we would then notice and fix.

[0] https://review.openstack.org/#/c/418928/

--Felipe

-----Original Message-----
From: Ghanshyam Mann [mailto:ghanshyammann@gmail.com]
Sent: Wednesday, April 19, 2017 12:05 AM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C fm577c@att.com wrote:
Hi all,

I have a question regarding patrole and release management. Many projects
like heat or murano have a tempest plugin within their repos, so by
extension their tempest plugins have releases, as the projects change over
time. However, since patrole is just a tempest plugin, yet heavily reliant
on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

Am I correct in
thinking that it should, in the first place? With nova-network and other
APIs slated for deprecation in Pike and beyond, Patrole will logically have
to continuously be maintained to keep up, meaning that older tests, just
like with Tempest, will have to be phased out. If Patrole, then, does not
have releases, then older release-dependent tests and functionality will
over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.

Thank You,

Felipe Monteiro

..1 https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.openstack.org_wiki_QA_releases&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=9eq8-o16PTMYfNYW0LjlVyRfN1W4wXWloI7jVij5W4g&e=
..2 https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_391113_13&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=A5axQVIlZlVMGaW0vwQxpqwx5uMZvZ1ZNwWAXgrIoAc&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e=

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 20, 2017 by MONTEIRO,_FELIPE_C (520 points)  
0 votes

On Fri, Apr 21, 2017 at 4:19 AM, MONTEIRO, FELIPE C fm577c@att.com wrote:
Thanks Ghanshyam,

That was really helpful. If Tempest is versioned via tags, then I agree that Patrole should do the same thing. I didn't quite know what you meant by "feature flag" but after finding [0] it became apparent to me: Patrole has been doing this wherever Tempest does it. Granted, we probably need more participation in the project to catch minute details like that, as we currently have a large body of tests, but currently we have been using feature flags where appropriate.

Very Nice. Feel free to ping us if you need any help on those things.

Case 3 as you mentioned is something that we've already had to account for: When Nova and then Keystone moved their policies into code, we had to add logic to the framework to account for those changes. Case 3 in theory would not be a problem, if there was a way to verify whether a policy action was enforced, other than by checking logs. If there was an API in OpenStack that could be called to retrieve oslo_policy enforcements per request id, then Case 3 would just result in an error being thrown by Patrole for a given test, which we would then notice and fix.

Yea that's true and it was due to lack of policy documentation. Nova
will (almost completed) have good documentation to solve that. John &
OSIC folks started very nice documentation on policy which tells you
what all policy being used by what all APIs with description [1]. And
on same page Keystone also [2]

This is how it looks like now [3]. This can help Patrole.

..1 https://blueprints.launchpad.net/nova/+spec/policy-docs
..2 https://blueprints.launchpad.net/keystone/+spec/policy-docs
..3 https://docs.openstack.org/developer/nova/sample_policy.html

[0] https://review.openstack.org/#/c/418928/

--Felipe

-----Original Message-----
From: Ghanshyam Mann [mailto:ghanshyammann@gmail.com]
Sent: Wednesday, April 19, 2017 12:05 AM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [qa] [patrole] Question regarding patrole release management

On Wed, Apr 19, 2017 at 2:29 AM, MONTEIRO, FELIPE C fm577c@att.com wrote:

Hi all,

I have a question regarding patrole and release management. Many projects
like heat or murano have a tempest plugin within their repos, so by
extension their tempest plugins have releases, as the projects change over
time. However, since patrole is just a tempest plugin, yet heavily reliant
on tempest, how should patrole do release management?

Hi Felipe,

Release management depends on whether Patrole is planning the
branchless model (i think it is) like Tempest or branch model., If
branchless, then it does not fall under release management. It can
adopt release model what Tempest has [1].

Branchless model gives many benefit like avoid backward incompatible
changes, avoid maintaining multiple Patrole repo across releases etc.

Am I correct in
thinking that it should, in the first place? With nova-network and other
APIs slated for deprecation in Pike and beyond, Patrole will logically have
to continuously be maintained to keep up, meaning that older tests, just
like with Tempest, will have to be phased out. If Patrole, then, does not
have releases, then older release-dependent tests and functionality will
over time be lost.

We have 3 cases here:
1. Functionality is deprecated/removed in new release.
2. Functionality newly added in new release.
3. Policy enforcement change.

For case 1, Tempest keep testing deprecated functionality till it is
marked deprecated across all supported stable branches. Once all
stable branch has that functionality as deprecated marked, then we
discuss of removing its testing from Tempest.
With API Microversion model that is little bit different where
functionality might be deprecated after specific version or it is
deprecated from base version itself. For example, Nova proxy APIs
deprecated after 2.36, Certificate APIs might be gone from base
version (which is under discussion). This is more case by case and
based on all stack holder point of view, we will decide their testing
should be removed or stay till when.

For case 2, Tempest introduced testing of new functionality with
feature flag and those tests will be executed/skipped accordingly.

Case 3 is something important to consider for Patrole. Usually policy
changes will be done with backward compatible way where changes does
not break upgrade. Any change in policy enforcement will be done at
least with supporting old and new rules [2] or with old rules
deprecation of period of 1 release cycle at least. And branchless
model can detect any accidental changes which going to break upgrade.

IMO, branchless model is good value for Patrole in all 3 cases
consideration and feature flags to handle the new/old/policy-change
functionality. Similarly release model can be same as Tempest has.

Thank You,

Felipe Monteiro

..1 https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.openstack.org_wiki_QA_releases&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=9eq8-o16PTMYfNYW0LjlVyRfN1W4wXWloI7jVij5W4g&e=
..2 https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_391113_13&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=A5axQVIlZlVMGaW0vwQxpqwx5uMZvZ1ZNwWAXgrIoAc&e=

-gmann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=xgobDX0RcKxlQhqhOSmU6H11-k8u21HsinBnbYLB6mY&s=DKOquEt2EsEYictxTecUKp5fwVaWZ3kNoFdm_tYFSsc&e=

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 21, 2017 by GHANSHYAM_MANN (5,700 points)   1 3 4
...