settingsLogin | Registersettings

[openstack-dev] [QA][blazar][ceilometer][congress][intel-nfv-ci-tests][ironic][manila][networking-bgpvpn][networking-fortinet][networking-sfc][neutron][neutron-fwaas][neutron-lbaas][nova-lxd][octavia][sahara][tap-as-a-service][horizon][vmware-nsx][watcher][all] Refactor of Tempest scenario base classes

0 votes

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor a smooth experience for our uses in OpenStack, and avoid painful
gate breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to change
your imports as a lot of projects use tempest/scenario in their code base.
You may decide to include the bare minimum you need from that module
instead of all of it. It's a bit more work to make the patch, but less
un-used code lying around afterwards.
- the QA team will refactor scenario tests, and make more interfaces stable
(test.py, credential providers). We won't advertise every single change in
this process, only when we start and once we're done.
- you may decide to discard your local copy of manager.py and consume
Tempest stable interfaces directly. We will help with any question you may
have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py

[1] https://etherpad.openstack.org/p/pike-qa-priorities
[2]
https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 24, 2017 in openstack-dev by Andrea_Frittoli (5,920 points)   1 2 3

12 Responses

0 votes

hi,

On Mon, Feb 27, 2017 at 8:34 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:
Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this refactor
a smooth experience for our uses in OpenStack, and avoid painful gate
breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to change
your imports as a lot of projects use tempest/scenario in their code base.
You may decide to include the bare minimum you need from that module instead
of all of it. It's a bit more work to make the patch, but less un-used code
lying around afterwards.

i submitted patches for a few repos.
https://review.openstack.org/#/q/status:open++branch:master+topic:tempest-manager
i'd suggest to use the same gerrit topic for relevant patches.

  • the QA team will refactor scenario tests, and make more interfaces stable
    (test.py, credential providers). We won't advertise every single change in
    this process, only when we start and once we're done.
  • you may decide to discard your local copy of manager.py and consume
    Tempest stable interfaces directly. We will help with any question you may
    have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py
[1] https://etherpad.openstack.org/p/pike-qa-priorities
[2]
https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by Takashi_Yamamoto (4,400 points)   2 3
0 votes

On Wed, Mar 1, 2017 at 2:21 AM Takashi Yamamoto yamamoto@midokura.com
wrote:

hi,

On Mon, Feb 27, 2017 at 8:34 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today
for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor
a smooth experience for our uses in OpenStack, and avoid painful gate
breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to
change
your imports as a lot of projects use tempest/scenario in their code
base.
You may decide to include the bare minimum you need from that module
instead
of all of it. It's a bit more work to make the patch, but less un-used
code
lying around afterwards.

i submitted patches for a few repos.

https://review.openstack.org/#/q/status:open++branch:master+topic:tempest-manager
i'd suggest to use the same gerrit topic for relevant patches.

Thank you for looking into this!
Having a common gerrit topic is a nice idea: "tempest-manager"

I'm also tracking patches in this etherpad:
https://etherpad.openstack.org/p/tempest-manager-plugins

andrea

  • the QA team will refactor scenario tests, and make more interfaces
    stable
    > (test.py, credential providers). We won't advertise every single change
    in
    > this process, only when we start and once we're done.
    > - you may decide to discard your local copy of manager.py and consume
    > Tempest stable interfaces directly. We will help with any question you
    may
    > have on the process and on Tempest interfaces.
    >
    > Repositories affected by the refactor are (based on [2]):
    >
    >
    blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher
    >
    > If we don't hear from a team at all in the next two weeks, we will assume
    > that the corresponding Tempest plugin / bunch of tests is not in use
    > anymore, and ignore it. If you use tempest.scenario.manager.py today and
    > your repo is not on the list, please let us know!
    >
    > I'm happy to propose an initial patch for any team that may require it -
    > just ping me on IRC (andreaf).
    > I won't have the bandwidth myself to babysit each patch through review
    and
    > gate though.
    >
    > Thank you for your cooperation and patience!
    >
    > Andrea
    >
    > [0]
    >
    http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py
    > [1] https://etherpad.openstack.org/p/pike-qa-priorities
    > [2]
    >
    https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh
    >
    >

    > OpenStack Development Mailing List (not for usage questions)
    > Unsubscribe:
    OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
    > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
    >


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Hi Andrea,

I am not sure why the new tempest.scenario.manager has to be developed that
way. May I humbly suggest another path? Roughly goes like:

1) Rename the present class to "OldScenarioTest", set the original name to
point to it (ScenarioTest=OldScenarioTest) (in a single commit)

2) Create a new class NewScenarioTest (initially a copy of the old one?),
development/refactoring/rewrite happens there, do not merge any new
features in OldScenarioTest

3) add a tempest config option "USENEWSCENARIO_MANAGER", False by default

4) conditionally on the value of USENEWSCENARIO_MANAGER set the
ScenarioTest to point to OldScenarioTest or NewScenarioTest

5) add a gate job(s) to tempest that forces USENEWSCENARIO_MANAGER to
True to test the new code, non-voting from the start, gate on it later

This way the projects using tempest plugins and other tempest consumers
will not be affected at all. Later each project on its jobs using tempest
can try to enforce USENEWSCENARIO_MANAGER to True if wishing to test the
new one or switch to it when it is ready. Eventually when new one is
stable, the config option might be removed and new class renamed to the
name of the original class.

The only downside of such approach I see might be a small? explosion of
number of jobs in the tempest project (running on both new and old
manager), but I'd think at least in the beginning of this refactoring
effort the tempest team would like to have the number of jobs testing the
new manager limited any way.

I might be completely wrong with this advice, as I presume tempest team
have put a lot of thinking into this problem already. But still, could you
consider such approach?

Cheers,

Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com

On Mon, Feb 27, 2017 at 1:34 PM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor a smooth experience for our uses in OpenStack, and avoid painful
gate breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to change
your imports as a lot of projects use tempest/scenario in their code base.
You may decide to include the bare minimum you need from that module
instead of all of it. It's a bit more work to make the patch, but less
un-used code lying around afterwards.
- the QA team will refactor scenario tests, and make more interfaces
stable (test.py, credential providers). We won't advertise every single
change in this process, only when we start and once we're done.
- you may decide to discard your local copy of manager.py and consume
Tempest stable interfaces directly. We will help with any question you may
have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,
manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,
neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-
service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review and
gate though.

Thank you for your cooperation and patience!

Andrea

[0] http://git.openstack.org/cgit/openstack/tempest/tree/
tempest/scenario/manager.py
[1] https://etherpad.openstack.org/p/pike-qa-priorities
[2] https://github.com/andreafrittoli/tempest_stable_
interfaces/blob/master/data/get_deps.sh


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2017 by Pavlo_Shchelokovskyy (4,760 points)   5 5
0 votes

Hi Pavlo,

thank you for your suggestion.
See my comments inline.

andrea
On Thu, Mar 2, 2017 at 9:19 AM Pavlo Shchelokovskyy <
pshchelokovskyy@mirantis.com> wrote:

Hi Andrea,

I am not sure why the new tempest.scenario.manager has to be developed
that way. May I humbly suggest another path? Roughly goes like:

1) Rename the present class to "OldScenarioTest", set the original name to
point to it (ScenarioTest=OldScenarioTest) (in a single commit)

We considered a similar option [0], but I fear that maintaining the old
class in Tempest, and even gating on it, would just postpone the issue for
plugins, and would probably have the negative effect of making even more
plugin rely on that.

2) Create a new class NewScenarioTest (initially a copy of the old one?),
development/refactoring/rewrite happens there, do not merge any new
features in OldScenarioTest

3) add a tempest config option "USENEWSCENARIO_MANAGER", False by default

4) conditionally on the value of USENEWSCENARIO_MANAGER set the
ScenarioTest to point to OldScenarioTest or NewScenarioTest

5) add a gate job(s) to tempest that forces USENEWSCENARIO_MANAGER to
True to test the new code, non-voting from the start, gate on it later

Having a gate job like this would make it really hard for us to get rid of
the it in future.

This way the projects using tempest plugins and other tempest consumers
will not be affected at all. Later each project on its jobs using tempest
can try to enforce USENEWSCENARIO_MANAGER to True if wishing to test the
new one or switch to it when it is ready. Eventually when new one is
stable, the config option might be removed and new class renamed to the
name of the original class.

The only downside of such approach I see might be a small? explosion of
number of jobs in the tempest project (running on both new and old
manager), but I'd think at least in the beginning of this refactoring
effort the tempest team would like to have the number of jobs testing the
new manager limited any way.

I might be completely wrong with this advice, as I presume tempest team
have put a lot of thinking into this problem already. But still, could you
consider such approach?

Another option would be to:
- create a new base class in Tempest
- incrementally move scenario tests across to the new base class, along
with the helpers they need
However this would have the drawback of risking to break plugins again and
again, and to slow down the overall process.

[0] https://review.openstack.org/#/c/439291/1

Cheers,

Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com

On Mon, Feb 27, 2017 at 1:34 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor a smooth experience for our uses in OpenStack, and avoid painful
gate breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to change
your imports as a lot of projects use tempest/scenario in their code base.
You may decide to include the bare minimum you need from that module
instead of all of it. It's a bit more work to make the patch, but less
un-used code lying around afterwards.
- the QA team will refactor scenario tests, and make more interfaces
stable (test.py, credential providers). We won't advertise every single
change in this process, only when we start and once we're done.
- you may decide to discard your local copy of manager.py and consume
Tempest stable interfaces directly. We will help with any question you may
have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py

[1] https://etherpad.openstack.org/p/pike-qa-priorities
[2]
https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 7, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Hi,

an update on this.

It's about 10days since the original message, and the current status is:
- 3 patches merged, 1 approved (recheck)
- 5 patches submitted, pending approval
- 2 patches with a -1 (need more work)
- 7 patches submitted by me today (draft) - review needed

Thank you for your work on this!

I would recommend to prune the imported module as much as possible as well.
It would make it easier for the QA team to identify which interfaces on
Tempest side should be migrated to stable.

andrea

On Wed, Mar 1, 2017 at 1:25 PM Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Wed, Mar 1, 2017 at 2:21 AM Takashi Yamamoto yamamoto@midokura.com
wrote:

hi,

On Mon, Feb 27, 2017 at 8:34 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today
for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor
a smooth experience for our uses in OpenStack, and avoid painful gate
breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to
change
your imports as a lot of projects use tempest/scenario in their code
base.
You may decide to include the bare minimum you need from that module
instead
of all of it. It's a bit more work to make the patch, but less un-used
code
lying around afterwards.

i submitted patches for a few repos.

https://review.openstack.org/#/q/status:open++branch:master+topic:tempest-manager
i'd suggest to use the same gerrit topic for relevant patches.

Thank you for looking into this!
Having a common gerrit topic is a nice idea: "tempest-manager"

I'm also tracking patches in this etherpad:
https://etherpad.openstack.org/p/tempest-manager-plugins

andrea

  • the QA team will refactor scenario tests, and make more interfaces
    stable
    (test.py, credential providers). We won't advertise every single change
    in
    this process, only when we start and once we're done.
  • you may decide to discard your local copy of manager.py and consume
    Tempest stable interfaces directly. We will help with any question you
    may
    have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review
and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py

https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 7, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On 02/27/2017 12:34 PM, Andrea Frittoli wrote:
Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please maintain a
copy of [0] in tree until further notice.

Hi!

I hope it is pretty obvious, but just to be clear. Anything that this copied
file uses should be treated more or less as a stable API by the QA team during
the whole transition period. The last thing we want to happen is for this file
to break all the time because its dependencies (imports, functions, classes it
uses) are not stable.

If it's not the case, please update it, and let us know the git hash to use to
grab the final version of the file.

Thanks for understanding!


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 15, 2017 by Dmitry_Tantsur (18,080 points)   2 3 7
0 votes

On Wed, Mar 15, 2017 at 11:38 AM Dmitry Tantsur dtantsur@redhat.com wrote:

On 02/27/2017 12:34 PM, Andrea Frittoli wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain a
copy of [0] in tree until further notice.

Hi!

I hope it is pretty obvious, but just to be clear. Anything that this
copied
file uses should be treated more or less as a stable API by the QA team
during
the whole transition period. The last thing we want to happen is for this
file
to break all the time because its dependencies (imports, functions,
classes it
uses) are not stable.

If it's not the case, please update it, and let us know the git hash to
use to
grab the final version of the file.

Your code depends on manager.py and its dependencies today,
and copying that in-tree removes at least one of the dependencies.

The only case were you're in a worse situation is if one of the imports is
removed / renamed,
and we'll do our best to avoid that.

My recommendation would be to trim down your copy of manager.py to the bare
minimum you
need, which is likely to be much smaller than the whole module.

Thanks for understanding!


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 15, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Hi.
We’ve completed integrating manager.py into openstack/ironic.
https://review.openstack.org/#/c/439252/ (include local copy)
https://review.openstack.org/#/c/446844/ (prune local copy)

-Solio

From: Andrea Frittoli andrea.frittoli@gmail.com
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Tuesday, March 7, 2017 at 3:28 PM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [QA][blazar][ceilometer][congress][intel-nfv-ci-tests][ironic][manila][networking-bgpvpn][networking-fortinet][networking-sfc][neutron][neutron-fwaas][neutron-lbaas][nova-lxd][octavia][sahara][tap-as-a-service][horizon][vmware-nsx][...

Hi,

an update on this.

It's about 10days since the original message, and the current status is:
- 3 patches merged, 1 approved (recheck)
- 5 patches submitted, pending approval
- 2 patches with a -1 (need more work)
- 7 patches submitted by me today (draft) - review needed

Thank you for your work on this!

I would recommend to prune the imported module as much as possible as well.
It would make it easier for the QA team to identify which interfaces on Tempest side should be migrated to stable.

andrea

On Wed, Mar 1, 2017 at 1:25 PM Andrea Frittoli andrea.frittoli@gmail.com wrote:
On Wed, Mar 1, 2017 at 2:21 AM Takashi Yamamoto yamamoto@midokura.com wrote:
hi,

On Mon, Feb 27, 2017 at 8:34 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:
Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this refactor
a smooth experience for our uses in OpenStack, and avoid painful gate
breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to change
your imports as a lot of projects use tempest/scenario in their code base.
You may decide to include the bare minimum you need from that module instead
of all of it. It's a bit more work to make the patch, but less un-used code
lying around afterwards.

i submitted patches for a few repos.
https://review.openstack.org/#/q/status:open++branch:master+topic:tempest-manager
i'd suggest to use the same gerrit topic for relevant patches.
Thank you for looking into this!
Having a common gerrit topic is a nice idea: "tempest-manager"

I'm also tracking patches in this etherpad: https://etherpad.openstack.org/p/tempest-manager-plugins

andrea

  • the QA team will refactor scenario tests, and make more interfaces stable
    (test.py, credential providers). We won't advertise every single change in
    this process, only when we start and once we're done.
  • you may decide to discard your local copy of manager.py and consume
    Tempest stable interfaces directly. We will help with any question you may
    have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]
http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py
[1] https://etherpad.openstack.org/p/pike-qa-priorities
[2]
https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 17, 2017 by Sarabia,_Solio (180 points)  
0 votes

On Fri, Mar 17, 2017 at 5:43 PM Sarabia, Solio solio.sarabia@intel.com
wrote:

Hi.

We’ve completed integrating manager.py into openstack/ironic.

https://review.openstack.org/#/c/439252/ (include local copy)

https://review.openstack.org/#/c/446844/ (prune local copy)

Thank you, much appreciated!

-Solio

*From: *Andrea Frittoli andrea.frittoli@gmail.com
*Reply-To: *"OpenStack Development Mailing List (not for usage
questions)" openstack-dev@lists.openstack.org
*Date: *Tuesday, March 7, 2017 at 3:28 PM
*To: *"OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
*Subject: *Re: [openstack-dev]
[QA][blazar][ceilometer][congress][intel-nfv-ci-tests][ironic][manila][networking-bgpvpn][networking-fortinet][networking-sfc][neutron][neutron-fwaas][neutron-lbaas][nova-lxd][octavia][sahara][tap-as-a-service][horizon][vmware-nsx][...

Hi,

an update on this.

It's about 10days since the original message, and the current status is:

  • 3 patches merged, 1 approved (recheck)

  • 5 patches submitted, pending approval

  • 2 patches with a -1 (need more work)

  • 7 patches submitted by me today (draft) - review needed

Thank you for your work on this!

I would recommend to prune the imported module as much as possible as well.

It would make it easier for the QA team to identify which interfaces on
Tempest side should be migrated to stable.

andrea

On Wed, Mar 1, 2017 at 1:25 PM Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Wed, Mar 1, 2017 at 2:21 AM Takashi Yamamoto yamamoto@midokura.com
wrote:

hi,

On Mon, Feb 27, 2017 at 8:34 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain
a copy of [0] in tree until further notice.

Full message:


One of the priorities for the QA team in the Pike cycle is to refactor
scenario tests to a sane code base [1].

As they are now, changes to scenario tests are difficult to develop and
review, and failures in those tests are hard to debug, which is in many
directions far away from where we need to be.

The issue we face is that, even though tempest.scenario.manager is not
advertised as a stable interface in Tempest, many project use it today
for
convenience in writing their own tests. We don't know about dependencies
outside of the OpenStack ecosystem, but we want to try to make this
refactor
a smooth experience for our uses in OpenStack, and avoid painful gate
breakages as much as possible.

The process we're proposing is as follows:
- hold a copy of [0] in tree - in most cases you won't even have to
change
your imports as a lot of projects use tempest/scenario in their code
base.
You may decide to include the bare minimum you need from that module
instead
of all of it. It's a bit more work to make the patch, but less un-used
code
lying around afterwards.

i submitted patches for a few repos.

https://review.openstack.org/#/q/status:open++branch:master+topic:tempest-manager
i'd suggest to use the same gerrit topic for relevant patches.

Thank you for looking into this!

Having a common gerrit topic is a nice idea: "tempest-manager"

I'm also tracking patches in this etherpad:
https://etherpad.openstack.org/p/tempest-manager-plugins

andrea

  • the QA team will refactor scenario tests, and make more interfaces
    stable
    (test.py, credential providers). We won't advertise every single change
    in
    this process, only when we start and once we're done.
  • you may decide to discard your local copy of manager.py and consume
    Tempest stable interfaces directly. We will help with any question you
    may
    have on the process and on Tempest interfaces.

Repositories affected by the refactor are (based on [2]):

blazar,ceilometer,congress,intel-nfv-ci-tests,ironic,manila,networking-bgpvpn,networking-fortinet,networking-sfc,neutron-fwaas,neutron-lbaas,nova-lxd,octavia,sahara-tests,tap-as-a-service,tempest-horizon,vmware-nsx,watcher

If we don't hear from a team at all in the next two weeks, we will assume
that the corresponding Tempest plugin / bunch of tests is not in use
anymore, and ignore it. If you use tempest.scenario.manager.py today and
your repo is not on the list, please let us know!

I'm happy to propose an initial patch for any team that may require it -
just ping me on IRC (andreaf).
I won't have the bandwidth myself to babysit each patch through review
and
gate though.

Thank you for your cooperation and patience!

Andrea

[0]

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py

https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 17, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Status update.

Thanks to all your work, we are getting close to finishing this work!

  • 12 patches merged
  • 7 patches ready, pending approval
  • 2 patches (on stable branches) with a pep8 failure to be fixed

Details: https://etherpad.openstack.org/p/tempest-manager-plugins

There are still a few patches that need a review, any help
would be much appreciated!

Thank you

andrea

On Wed, Mar 15, 2017 at 3:56 PM Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Wed, Mar 15, 2017 at 11:38 AM Dmitry Tantsur dtantsur@redhat.com
wrote:

On 02/27/2017 12:34 PM, Andrea Frittoli wrote:

Hello folks,

TL;DR: if today you import manager,py from tempest.scenario please
maintain a
copy of [0] in tree until further notice.

Hi!

I hope it is pretty obvious, but just to be clear. Anything that this
copied
file uses should be treated more or less as a stable API by the QA team
during
the whole transition period. The last thing we want to happen is for this
file
to break all the time because its dependencies (imports, functions,
classes it
uses) are not stable.

If it's not the case, please update it, and let us know the git hash to
use to
grab the final version of the file.

Your code depends on manager.py and its dependencies today,
and copying that in-tree removes at least one of the dependencies.

The only case were you're in a worse situation is if one of the imports is
removed / renamed,
and we'll do our best to avoid that.

My recommendation would be to trim down your copy of manager.py to the
bare minimum you
need, which is likely to be much smaller than the whole module.

Thanks for understanding!


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 18, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
...