settingsLogin | Registersettings

[openstack-dev] [QA]Refactoring Scenarios manager.py

0 votes

Hi guys,
So I have a problem with these 2 patches here 1 and here 2. You
basically are blocking any attempt of refactoring manager.py. Refactoring
that file has been our number one priority for 2 cycles, and so far hardly
no one stepped up really to do the work, except me with these 2 patches.
Let me remind you that that file is a gigantic mess, an so are our network
scenarios.

The manager.py file in the scenarios directory has no stable interface, and
it was never "advertised" so. That some plugins decided to use some private
methods (such as this getnetworkbyname) is unfortunate but that should
not block us from moving.

So just to be clear, if we really want to refactor our scenarios (and we
must in my opinion), things will break for projects that are importing
Tempest and using it outside of its stable interface. I am not interested
in being the good Samaritan for the whole OpenStack galaxy, I have enough
with the 6 core projects and the Tempest stable interface. So guys, if you
are and don't want to go forward with 1 and 2, be sure I'll never touch
those scenarios again. I am not upset, but we have to make clear decisions,
sometimes difficult.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 1, 2017 in openstack-dev by Jordan_Pittier (3,060 points)   2 3

15 Responses

0 votes

Hi,

Thank you for bringing this up.

Yeah, I understand your frustration. We already have the document about our
stable interface[1]. It says


Stability
Any code that lives in tempest/lib will be treated as a stable interface.
This means that any public interface under the tempest/lib directory is
expected to be a stable interface suitable for public consumption. However,
for any interfaces outside of tempest/lib in the tempest tree (unless
otherwise noted) or any private interfaces the same stability guarantees
don't apply.

So we can change private interfaces basically. However, I also assume that
this document is not well known(or people just ignore it, though, maybe).
So I'd like to advertise this policy here, and discuss it (if the
discussion is needed.)

[1] https://docs.openstack.org/developer/tempest/library.html#stability

On Sat, Feb 25, 2017 at 22:39 Jordan Pittier jordan.pittier@scality.com
wrote:

Hi guys,
So I have a problem with these 2 patches here [1] and here [2]. You
basically are blocking any attempt of refactoring manager.py. Refactoring
that file has been our number one priority for 2 cycles, and so far hardly
no one stepped up really to do the work, except me with these 2 patches.
Let me remind you that that file is a gigantic mess, an so are our network
scenarios.

The manager.py file in the scenarios directory has no stable interface,
and it was never "advertised" so. That some plugins decided to use some
private methods (such as this getnetworkbyname) is unfortunate but that
should not block us from moving.

So just to be clear, if we really want to refactor our scenarios (and we
must in my opinion), things will break for projects that are importing
Tempest and using it outside of its stable interface. I am not interested
in being the good Samaritan for the whole OpenStack galaxy, I have enough
with the 6 core projects and the Tempest stable interface. So guys, if you
are and don't want to go forward with [1] and [2], be sure I'll never touch
those scenarios again. I am not upset, but we have to make clear decisions,
sometimes difficult.

[1] : https://review.openstack.org/#/c/436555/
[2] : https://review.openstack.org/#/c/438097/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Masayuki Igawa


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 26, 2017 by masayuki_at_igawa.me (700 points)  
0 votes

On Sun, Feb 26, 2017 at 12:49 AM Masayuki Igawa masayuki@igawa.me wrote:

Hi,

Thank you for bringing this up.

Yeah, I understand your frustration. We already have the document about
our stable interface[1]. It says


Stability
Any code that lives in tempest/lib will be treated as a stable interface.
This means that any public interface under the tempest/lib directory is
expected to be a stable interface suitable for public consumption. However,
for any interfaces outside of tempest/lib in the tempest tree (unless
otherwise noted) or any private interfaces the same stability guarantees
don't apply.

So we can change private interfaces basically. However, I also assume that
this document is not well known(or people just ignore it, though, maybe).
So I'd like to advertise this policy here, and discuss it (if the
discussion is needed.)

[1] https://docs.openstack.org/developer/tempest/library.html#stability

On Sat, Feb 25, 2017 at 22:39 Jordan Pittier jordan.pittier@scality.com
wrote:

Hi guys,
So I have a problem with these 2 patches here [1] and here [2]. You
basically are blocking any attempt of refactoring manager.py. Refactoring
that file has been our number one priority for 2 cycles, and so far hardly
no one stepped up really to do the work, except me with these 2 patches.
Let me remind you that that file is a gigantic mess, an so are our network
scenarios.

The manager.py file in the scenarios directory has no stable interface,
and it was never "advertised" so. That some plugins decided to use some
private methods (such as this getnetworkbyname) is unfortunate but that
should not block us from moving.

I agree this should not block us from moving, and as you mentioned we
definitely need to move and I appreciate the fact that you started the work!

So just to be clear, if we really want to refactor our scenarios (and we
must in my opinion), things will break for projects that are importing
Tempest and using it outside of its stable interface. I am not interested
in being the good Samaritan for the whole OpenStack galaxy, I have enough
with the 6 core projects and the Tempest stable interface. So guys, if you
are and don't want to go forward with [1] and [2], be sure I'll never touch
those scenarios again. I am not upset, but we have to make clear decisions,
sometimes difficult.

We have no way to know exactly who's using unstable interfaces in Tempest,
so it's likely someone will have to change their code as a consequence of
the refactor.
But I think it's important that we are good citizens and advertise well
what's going to change, even if it's about an interface which is not
declared as stable.

[1] : https://review.openstack.org/#/c/436555/
[2] : https://review.openstack.org/#/c/438097/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Masayuki Igawa


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Scenario tests will go through a significant number of changes as part of
the refactor and if every change risks to break someone's gate job it won't
work.
I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to happen
- help the affected teams to identify a way decouple them from tempest
scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not merge
them yet

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail
tomorrow.

Andrea


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 26, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Yea, there is no doubt we should refactor scenario tests but even those are
internal interface it breaks plugins. We can argue that plugins should not
be using those but before that tempest should have all required
interface/class/helper as stable interface for plugins. which is what
scenario tests refactoring is trying to do.

​​I agree with the process Andrea defined and we should follow the same. If
we can put a deadline for projects to fix, we can speed up our work of
refactoring. I propose to keep refactoring patch for 2 week (including
comments fixes etc) and give time to plugins to fix and yes we will help
them. ​
After 2 week of time, we do not guarantee about any plugin failure (with
very rare exception if interface is being used very widely)

Let's not break plugins (what we doing as max as possible) but we really
need each plugins helps on those. QA team fix plugins since starting and we
have submitted lot of patches for many plugins to fix them and many of them
never got attention or reviewed.
For such cases (which falls under 2 week of deadlines), yes plugins needs
to take full responsibility if any of the tempest interface break them.

-gmann

On Mon, Feb 27, 2017 at 3:13 AM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Sun, Feb 26, 2017 at 12:49 AM Masayuki Igawa masayuki@igawa.me wrote:

Hi,

Thank you for bringing this up.

Yeah, I understand your frustration. We already have the document about
our stable interface[1]. It says


Stability
Any code that lives in tempest/lib will be treated as a stable interface.
This means that any public interface under the tempest/lib directory is
expected to be a stable interface suitable for public consumption. However,
for any interfaces outside of tempest/lib in the tempest tree (unless
otherwise noted) or any private interfaces the same stability guarantees
don't apply.

So we can change private interfaces basically. However, I also assume
that this document is not well known(or people just ignore it, though,
maybe). So I'd like to advertise this policy here, and discuss it (if the
discussion is needed.)

[1] https://docs.openstack.org/developer/tempest/library.html#stability

On Sat, Feb 25, 2017 at 22:39 Jordan Pittier jordan.pittier@scality.com
wrote:

Hi guys,
So I have a problem with these 2 patches here [1] and here [2]. You
basically are blocking any attempt of refactoring manager.py. Refactoring
that file has been our number one priority for 2 cycles, and so far hardly
no one stepped up really to do the work, except me with these 2 patches.
Let me remind you that that file is a gigantic mess, an so are our network
scenarios.

The manager.py file in the scenarios directory has no stable interface,
and it was never "advertised" so. That some plugins decided to use some
private methods (such as this getnetworkbyname) is unfortunate but that
should not block us from moving.

I agree this should not block us from moving, and as you mentioned we
definitely need to move and I appreciate the fact that you started the work!

So just to be clear, if we really want to refactor our scenarios (and we
must in my opinion), things will break for projects that are importing
Tempest and using it outside of its stable interface. I am not interested
in being the good Samaritan for the whole OpenStack galaxy, I have enough
with the 6 core projects and the Tempest stable interface. So guys, if you
are and don't want to go forward with [1] and [2], be sure I'll never touch
those scenarios again. I am not upset, but we have to make clear decisions,
sometimes difficult.

We have no way to know exactly who's using unstable interfaces in Tempest,
so it's likely someone will have to change their code as a consequence of
the refactor.
But I think it's important that we are good citizens and advertise well
what's going to change, even if it's about an interface which is not
declared as stable.

[1] : https://review.openstack.org/#/c/436555/
[2] : https://review.openstack.org/#/c/438097/



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Masayuki Igawa



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Scenario tests will go through a significant number of changes as part of
the refactor and if every change risks to break someone's gate job it won't
work.
I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to happen
- help the affected teams to identify a way decouple them from tempest
scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not
merge them yet​

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail
tomorrow.

Andrea


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 27, 2017 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

First I really appreciate Jordan's work, and I always appreciate those who really do something. If we don't have a begining, then we will never reach the end.

I now sort out the problem as:

1) Do we need to refactory the Tempest scenario code? ------ yes

2) What if scenario refactory will affect many plugins which use scenario's unstable code?

2.1) choice 1: plugins modify their code simply

       * for simplified modification, one week is enough

       * if plugins do nothing, then Tempest will not be responsible for their gate failures after one week.

2.2) choice 2: scenario provides a set of "stable interfaces", and plugins switch to use them.

       * if many plugins use interfaces in scenario, then it may worth to have a stable base of scenario(not the current manager.py)

       * it will need more work in Tempest, but plugins will benefit from it.

to sum up, Tempest should let plugins feel as comfortable as we can, but if plugins doing nothing no matter what we do,

then they won't stop the pace of our scenario refactoring.

Zhufl

Original Mail

Sender: <ghanshyammann@gmail.com>
To: <openstack-dev@lists.openstack.org>
Date: 2017/02/27 08:32
Subject: Re: [openstack-dev] [QA]Refactoring Scenarios manager.py

Yea, there is no doubt we should refactor scenario tests but even those are internal interface it breaks plugins. We can argue that plugins should not be using those but before that tempest should have all required interface/class/helper as stable interface for plugins. which is what scenario tests refactoring is trying to do.

I agree with the process Andrea defined and we should follow the same.. If we can put a deadline for projects to fix, we can speed up our work of refactoring. I propose to keep refactoring patch for 2 week (including comments fixes etc) and give time to plugins to fix and yes we will help them.

After 2 week of time, we do not guarantee about any plugin failure (with very rare exception if interface is being used very widely)

Let's not break plugins (what we doing as max as possible) but we really need each plugins helps on those. QA team fix plugins since starting and we have submitted lot of patches for many plugins to fix them and many of them never got attention or reviewed.

For such cases (which falls under 2 week of deadlines), yes plugins needs to take full responsibility if any of the tempest interface break them.

-gmann

On Mon, Feb 27, 2017 at 3:13 AM, Andrea Frittoli <andrea.frittoli@gmail.com> wrote:

On Sun, Feb 26, 2017 at 12:49 AM Masayuki Igawa <masayuki@igawa.me> wrote:

Hi,Thank you for bringing this up.Yeah, I understand your frustration. We already have the document about our stable interface[1]. It says ------StabilityAny code that lives in tempest/lib will be treated as a stable interface. This means that any public interface under the tempest/lib directory is expected to be a stable interface suitable for public consumption. However, for any interfaces outside of tempest/lib in the tempest tree (unless otherwise noted) or any private interfaces the same stability guarantees don't apply.------So we can change private interfaces basically. However, I also assume that this document is not well known(or people just ignore it, though, maybe). So I'd like to advertise this policy here, and discuss it (if the discussion is needed.)

[1] https://docs.openstack.org/developer/tempest/library.html#stability

On Sat, Feb 25, 2017 at 22:39 Jordan Pittier <jordan.pittier@scality.com> wrote:

Hi guys,
So I have a problem with these 2 patches here [1] and here [2]. You basically are blocking any attempt of refactoring manager.py. Refactoring that file has been our number one priority for 2 cycles, and so far hardly no one stepped up really to do the work, except me with these 2 patches. Let me remind you that that file is a gigantic mess, an so are our network scenarios.

The manager.py file in the scenarios directory has no stable interface, and it was never "advertised" so. That some plugins decided to use some private methods (such as this getnetworkbyname) is unfortunate but that should not block us from moving.

I agree this should not block us from moving, and as you mentioned we definitely need to move and I appreciate the fact that you started the work!

So just to be clear, if we really want to refactor our scenarios (and we must in my opinion), things will break for projects that are importing Tempest and using it outside of its stable interface. I am not interested in being the good Samaritan for the whole OpenStack galaxy, I have enough with the 6 core projects and the Tempest stable interface. So guys, if you are and don't want to go forward with [1] and [2], be sure I'll never touch those scenarios again. I am not upset, but we have to make clear decisions, sometimes difficult.

We have no way to know exactly who's using unstable interfaces in Tempest, so it's likely someone will have to change their code as a consequence of the refactor.
But I think it's important that we are good citizens and advertise well what's going to change, even if it's about an interface which is not declared as stable.

[1] : https://review.openstack.org/#/c/436555/

[2] : https://review.openstack.org/#/c/438097/

__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Masayuki Igawa

__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Scenario tests will go through a significant number of changes as part of the refactor and if every change risks to break someone's gate job it won't work.

I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to happen
- help the affected teams to identify a way decouple them from tempest scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not merge them yet

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail tomorrow.

Andrea


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 27, 2017 by zhu.fanglei_at_zte.c (500 points)  
0 votes

On Mon, 27 Feb 2017, 12:32 a.m. Ghanshyam Mann, ghanshyammann@gmail.com
wrote:

Yea, there is no doubt we should refactor scenario tests but even those
are internal interface it breaks plugins. We can argue that plugins should
not be using those but before that tempest should have all required
interface/class/helper as stable interface for plugins. which is what
scenario tests refactoring is trying to do.

​​I agree with the process Andrea defined and we should follow the same.
If we can put a deadline for projects to fix, we can speed up our work of
refactoring. I propose to keep refactoring patch for 2 week (including
comments fixes etc) and give time to plugins to fix and yes we will help
them. ​
After 2 week of time, we do not guarantee about any plugin failure (with
very rare exception if interface is being used very widely)

I didn't want to an exact time frame in my message because I would say it
depends on a case by case.

Let's not break plugins (what we doing as max as possible) but we really
need each plugins helps on those. QA team fix plugins since starting and we
have submitted lot of patches for many plugins to fix them and many of them
never got attention or reviewed.
For such cases (which falls under 2 week of deadlines), yes plugins needs
to take full responsibility if any of the tempest interface break them.

I don't think this will work if we do it on a patch by patch basis. It
would slow us down too much and it would become an ongoing effort for other
teams which is probably not sustainable.

-gmann

On Mon, Feb 27, 2017 at 3:13 AM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

On Sun, Feb 26, 2017 at 12:49 AM Masayuki Igawa masayuki@igawa.me wrote:

Hi,

Thank you for bringing this up.

Yeah, I understand your frustration. We already have the document about
our stable interface[1]. It says


Stability
Any code that lives in tempest/lib will be treated as a stable interface.
This means that any public interface under the tempest/lib directory is
expected to be a stable interface suitable for public consumption. However,
for any interfaces outside of tempest/lib in the tempest tree (unless
otherwise noted) or any private interfaces the same stability guarantees
don't apply.

So we can change private interfaces basically. However, I also assume that
this document is not well known(or people just ignore it, though, maybe).
So I'd like to advertise this policy here, and discuss it (if the
discussion is needed.)

[1] https://docs.openstack.org/developer/tempest/library.html#stability

On Sat, Feb 25, 2017 at 22:39 Jordan Pittier jordan.pittier@scality.com
wrote:

Hi guys,
So I have a problem with these 2 patches here [1] and here [2]. You
basically are blocking any attempt of refactoring manager.py. Refactoring
that file has been our number one priority for 2 cycles, and so far hardly
no one stepped up really to do the work, except me with these 2 patches.
Let me remind you that that file is a gigantic mess, an so are our network
scenarios.

The manager.py file in the scenarios directory has no stable interface,
and it was never "advertised" so. That some plugins decided to use some
private methods (such as this getnetworkbyname) is unfortunate but that
should not block us from moving.

I agree this should not block us from moving, and as you mentioned we
definitely need to move and I appreciate the fact that you started the work!

So just to be clear, if we really want to refactor our scenarios (and we
must in my opinion), things will break for projects that are importing
Tempest and using it outside of its stable interface. I am not interested
in being the good Samaritan for the whole OpenStack galaxy, I have enough
with the 6 core projects and the Tempest stable interface. So guys, if you
are and don't want to go forward with [1] and [2], be sure I'll never touch
those scenarios again. I am not upset, but we have to make clear decisions,
sometimes difficult.

We have no way to know exactly who's using unstable interfaces in Tempest,
so it's likely someone will have to change their code as a consequence of
the refactor.
But I think it's important that we are good citizens and advertise well
what's going to change, even if it's about an interface which is not
declared as stable.

[1] : https://review.openstack.org/#/c/436555/
[2] : https://review.openstack.org/#/c/438097/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Masayuki Igawa


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Scenario tests will go through a significant number of changes as part of
the refactor and if every change risks to break someone's gate job it won't
work.
I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to happen
- help the affected teams to identify a way decouple them from tempest
scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not
merge them yet​

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail
tomorrow.

Andrea


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 27, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

I see Jordan's opinion here and I also faced this situation before.
I proposed a hacking patch 1 to notify wrong usage of Tempest
methods to projects and I saw some users of these methods didn't know
the definition of stable interfaces of Tempest.
We always face this issue on developments which change internal
methods of Tempest.

2017-02-26 10:13 GMT-08:00 Andrea Frittoli andrea.frittoli@gmail.com:

Scenario tests will go through a significant number of changes as part of
the refactor and if every change risks to break someone's gate job it won't
work.
I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to happen
- help the affected teams to identify a way decouple them from tempest
scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not merge
them yet

Yeah, this approach is very nice to land patches softly.
Users can find solutions if facing this issue on the own gate.

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail
tomorrow.

++, thanks for your leadership.

Thanks
Ken Ohmichi



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 27, 2017 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Doing gradual refactoring and fixing plugins time to time needs lot of wait
and sync.

That needs:
1. Plugins to switch from current method usage. Plugins to have some other
function or same copy paste code what current scenario base class has.
2. Tempest patch to wait for plugin fix.
3. Plugins to switch back to stable interface once Tempest going to provide
those.

This needs lot of sync between tempest and plugins and we have to wait for
tempest refactoring patch for long.

To make it more efficient, how about this:
1. Keep the scenario manger copy in tempest as it is. for plugins usage
only.
2. Start refactoring the scenario framework by adding more and more helper
function under /common or lib.
3. Once we feel all needed by scenario tests are available as stable
helper, then ask each plugins to switch to stable interface.
4. After all helpers are available, with deprecation period of 1 cycle
remove the scenario base class.

​Other way can be have copy of scenario manager or used interfaces in each
plugins (as mentioned by Andrea) but that needs changes in almost all
plugins and slow down Tempest work.​

​Have a look in this patch- https://review.openstack.org/#/c/439291/

-gmann

On Tue, Feb 28, 2017 at 3:28 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

I see Jordan's opinion here and I also faced this situation before.
I proposed a hacking patch [1] to notify wrong usage of Tempest
methods to projects and I saw some users of these methods didn't know
the definition of stable interfaces of Tempest.
We always face this issue on developments which change internal
methods of Tempest.

2017-02-26 10:13 GMT-08:00 Andrea Frittoli andrea.frittoli@gmail.com:

Scenario tests will go through a significant number of changes as part of
the refactor and if every change risks to break someone's gate job it
won't
work.
I propose we proceed as follows:
- identify projects that import from tempest.scenario
- send a notification to the ML about the changes that are going to
happen
- help the affected teams to identify a way decouple them from tempest
scenario code - most likely copy the relevant code in-tree
- meanwhile continue to work on patches for scenario tests but do not
merge
them yet

Yeah, this approach is very nice to land patches softly.
Users can find solutions if facing this issue on the own gate.

This process shouldn't take long and be rather straight forward.

I already have some data from codesearch, I will send out the e-mail
tomorrow.

++, thanks for your leadership.

Thanks
Ken Ohmichi


[1]: https://review.openstack.org/#/c/328651


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

I think it a good solution, I already put +1 :)

And, as to the scenario testcases, shall we:

1) remove test steps/checks already coverd in API test

2) remove sequence test cases (such as testserversequencesuspendresume), othersize scenario will get fatter and fatter

Original Mail

Sender: <ghanshyammann@gmail.com>
To: <openstack-dev@lists.openstack.org>
Date: 2017/03/01 11:03
Subject: Re: [openstack-dev] [QA]Refactoring Scenarios manager.py

Doing gradual refactoring and fixing plugins time to time needs lot of wait and sync.

That needs:

  1. Plugins to switch from current method usage. Plugins to have some other function or same copy paste code what current scenario base class has.

  2. Tempest patch to wait for plugin fix.

  3. Plugins to switch back to stable interface once Tempest going to provide those.

This needs lot of sync between tempest and plugins and we have to wait for tempest refactoring patch for long.

To make it more efficient, how about this:

  1. Keep the scenario manger copy in tempest as it is. for plugins usage only.

  2. Start refactoring the scenario framework by adding more and more helper function under /common or lib.

  3. Once we feel all needed by scenario tests are available as stable helper, then ask each plugins to switch to stable interface.

  4. After all helpers are available, with deprecation period of 1 cycle remove the scenario base class.

Other way can be have copy of scenario manager or used interfaces in each plugins (as mentioned by Andrea) but that needs changes in almost all plugins and slow down Tempest work.

Have a look in this patch- https://review.openstack.org/#/c/439291/

-gmann

On Tue, Feb 28, 2017 at 3:28 AM, Ken'ichi Ohmichi <ken1ohmichi@gmail.com> wrote:
I see Jordan's opinion here and I also faced this situation before.
I proposed a hacking patch [1] to notify wrong usage of Tempest
methods to projects and I saw some users of these methods didn't know
the definition of stable interfaces of Tempest.
We always face this issue on developments which change internal
methods of Tempest.

2017-02-26 10:13 GMT-08:00 Andrea Frittoli <andrea.frittoli@gmail.com>:

> Scenario tests will go through a significant number of changes as part of
> the refactor and if every change risks to break someone's gate job it won't
> work.
> I propose we proceed as follows:
> - identify projects that import from tempest.scenario
> - send a notification to the ML about the changes that are going to happen
> - help the affected teams to identify a way decouple them from tempest
> scenario code - most likely copy the relevant code in-tree
> - meanwhile continue to work on patches for scenario tests but do not merge
> them yet

Yeah, this approach is very nice to land patches softly.
Users can find solutions if facing this issue on the own gate.

> This process shouldn't take long and be rather straight forward.

> I already have some data from codesearch, I will send out the e-mail
> tomorrow.

++, thanks for your leadership.

Thanks
Ken Ohmichi


[1]: https://review.openstack.org/#/c/328651


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by zhu.fanglei_at_zte.c (500 points)  
0 votes

On Wed, Mar 1, 2017 at 3:57 AM, Ghanshyam Mann ghanshyammann@gmail.com
wrote:

Doing gradual refactoring and fixing plugins time to time needs lot of
wait and sync.

That needs:
1. Plugins to switch from current method usage. Plugins to have some other
function or same copy paste code what current scenario base class has.
2. Tempest patch to wait for plugin fix.
3. Plugins to switch back to stable interface once Tempest going to
provide those.

This needs lot of sync between tempest and plugins and we have to wait for
tempest refactoring patch for long.

To make it more efficient, how about this:
1. Keep the scenario manger copy in tempest as it is. for plugins usage
only.

Given that the refactoring effort "started" a year ago, at the current
speed it may take 2 years to complete. In the mean time we will have a
massive code duplication and a maintenance burden.

  1. Start refactoring the scenario framework by adding more and more helper
    function under /common or lib.

Starting a "framework" (each time I see that word, I have a bad feeling)
from scratch without users and usage is very very difficult. How do we know
what we need in that framework and what will be actually used in tests ?

The effort was called scenario refactoring and I think that's what we
should do. We should not do "start from scratch scenarios" or "copy all the
code and see what happens".

There's no problem with plugins. We committed to have a stable interface
which is documented and agreed upon. It's clearly written here
https://docs.openstack.org/developer/tempest/plugin.html#stable-tempest-apis-plugins-may-use
The rest is private, for Tempest internal use. If Tempest cores disagree
with that, then we should first of all put a spec and rewrite that
document.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by Jordan_Pittier (3,060 points)   2 3
0 votes

On Wed, Mar 1, 2017 at 4:18 AM, zhu.fanglei@zte.com.cn wrote:

I think it a good solution, I already put +1 :)

And, as to the scenario testcases, shall we:

1) remove test steps/checks already coverd in API test

Duplicate test steps/checks is not good and should be removed. It's not
related to the scenario refactoring effort, so please if you find
duplicated tests or test steps, we should remove them.

2) remove sequence test cases (such as testserversequencesuspendresume),
othersize scenario will get fatter and fatter

There's no definitive answer to that. We should just remember what a
scenario should be: test several openstack components, "real" world use
cases, "real" integration testing. Those should be our guideline for
scenarios. We should not buy into "I put it into the scenarios directory
because the helper methods were here and convenient" or "because I saw an
already existing scenario that look kind of the same".


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 1, 2017 by Jordan_Pittier (3,060 points)   2 3
...