settingsLogin | Registersettings

[openstack-dev] [QA] The future of nosetests with Tempest

0 votes

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with nosetests is going
away. We've started using newer features that nose just doesn't support. One
example of this is that we've started using testscenarios and we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove nosetests from the
requirements list and all the workarounds and references to nose will be pulled
out of the tree. Tempest will also start raising an unsupported exception when
you try to run it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've brought it up a
couple of times before, but I believe it is time to do this now. I feel for
tempest to move forward we need to do this now so that there isn't any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with python 2.6
since up until now we've set nosetests. There is a workaround for getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just doesn't
work with tempest, and we're adding more features to tempest, your negative test
generator being one of them, that interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario tests
every day with nose on RHEL 6 with minimal issues. I had to workaround
the negative test discovery by simply sed'ing that out of the tests
before running it, but that's acceptable to me until we can start
testing on RHEL 7. Otherwise I'm completely OK with saying py26 isn't
really supported and isn't used in the gate, and it's a buyer beware
situation to make it work, which includes pushing up trivial patches to
make it work (which I did a few of last week, and they were small syntax
changes or usages of testtools).

I don't understand how the core projects can be running unit tests in
the gate on py26 but our functional integration project is going to
actively go out and make it harder to run Tempest with py26, that sucks.

If we really want to move the test project away from py26, let's make
the concerted effort to get the core projects to move with it.

And FWIW, I tried the discover.py patch with unittest2 and testscenarios
last week and either I botched it, it's not documented properly on how
to apply it, or I screwed something up, but it didn't work for me, so
I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6 in
mind)? Using tox with testr and pip? I'm doing this all single-node.

patches this cycle that attempted to introduce incorrect behavior while trying
to fix compatibility with nose. That's why I think we need a clear message on
this sooner than later. Which is why I'm proposing actively raising an error
when things are run with nose upfront so there isn't any illusion that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile category.
Nose support is independent of python 2.6 support. Py26 I would still consider
to be unsupported, the issue is that the hack to make py26 work is outside of
tempest. This is why we've recommended that people using python 2.6 run with
nose, which really is no longer an option. Attila's abandoned patch that I
linked above documents points to this bug with a patch to discover which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79

-Matt Treinish


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann

asked Feb 12, 2014 in openstack-dev by Matt_Riedemann (48,320 points)   3 11 31
retagged Feb 25, 2015 by admin

8 Responses

0 votes

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with nosetests is going
away. We've started using newer features that nose just doesn't support. One
example of this is that we've started using testscenarios and we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove nosetests from the
requirements list and all the workarounds and references to nose will be pulled
out of the tree. Tempest will also start raising an unsupported exception when
you try to run it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've brought it up a
couple of times before, but I believe it is time to do this now. I feel for
tempest to move forward we need to do this now so that there isn't any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with python 2.6
since up until now we've set nosetests. There is a workaround for getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just doesn't
work with tempest, and we're adding more features to tempest, your negative test
generator being one of them, that interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same after this
change. The only thing that we'll be doing is actively preventing nose from
working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling nose the
only path to run tempest with py2.6 is to use testr. (which is what it always
should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs to be done
is to apply that discover patch:

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6. (The only
difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior while trying
to fix compatibility with nose. That's why I think we need a clear message on
this sooner than later. Which is why I'm proposing actively raising an error
when things are run with nose upfront so there isn't any illusion that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile category.
Nose support is independent of python 2.6 support. Py26 I would still consider
to be unsupported, the issue is that the hack to make py26 work is outside of
tempest. This is why we've recommended that people using python 2.6 run with
nose, which really is no longer an option. Attila's abandoned patch that I
linked above documents points to this bug with a patch to discover which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79

responded Feb 12, 2014 by Matthew_Treinish (11,200 points)   2 5 6
0 votes

On 2/12/2014 1:57 PM, Matthew Treinish wrote:
On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with nosetests is going
away. We've started using newer features that nose just doesn't support. One
example of this is that we've started using testscenarios and we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove nosetests from the
requirements list and all the workarounds and references to nose will be pulled
out of the tree. Tempest will also start raising an unsupported exception when
you try to run it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've brought it up a
couple of times before, but I believe it is time to do this now. I feel for
tempest to move forward we need to do this now so that there isn't any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with python 2.6
since up until now we've set nosetests. There is a workaround for getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just doesn't
work with tempest, and we're adding more features to tempest, your negative test
generator being one of them, that interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same after this
change. The only thing that we'll be doing is actively preventing nose from
working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling nose the
only path to run tempest with py2.6 is to use testr. (which is what it always
should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs to be done
is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6. (The only
difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior while trying
to fix compatibility with nose. That's why I think we need a clear message on
this sooner than later. Which is why I'm proposing actively raising an error
when things are run with nose upfront so there isn't any illusion that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile category.
Nose support is independent of python 2.6 support. Py26 I would still consider
to be unsupported, the issue is that the hack to make py26 work is outside of
tempest. This is why we've recommended that people using python 2.6 run with
nose, which really is no longer an option. Attila's abandoned patch that I
linked above documents points to this bug with a patch to discover which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that didn't
work with the PowerVM driver in Nova but I'm not aware of something
similar with testr.

--

Thanks,

Matt Riedemann

responded Feb 26, 2014 by Matt_Riedemann (48,320 points)   3 11 31
0 votes

On 2/25/2014 7:46 PM, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with
nosetests is going
away. We've started using newer features that nose just doesn't
support. One
example of this is that we've started using testscenarios and
we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the
requirements list and all the workarounds and references to nose
will be pulled
out of the tree. Tempest will also start raising an unsupported
exception when
you try to run it with nose so that there isn't any confusion on
this moving
forward. We talked about doing this at summit briefly and I've
brought it up a
couple of times before, but I believe it is time to do this now. I
feel for
tempest to move forward we need to do this now so that there isn't
any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with
python 2.6
since up until now we've set nosetests. There is a workaround for
getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported
on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it
basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just
doesn't
work with tempest, and we're adding more features to tempest, your
negative test
generator being one of them, that interfere further with nose. I've
seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same
after this
change. The only thing that we'll be doing is actively preventing nose
from
working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling
nose the
only path to run tempest with py2.6 is to use testr. (which is what it
always
should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in
the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs
to be done
is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6.
(The only
difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior
while trying
to fix compatibility with nose. That's why I think we need a clear
message on
this sooner than later. Which is why I'm proposing actively raising
an error
when things are run with nose upfront so there isn't any illusion
that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
Nose support is independent of python 2.6 support. Py26 I would
still consider
to be unsupported, the issue is that the hack to make py26 work is
outside of
tempest. This is why we've recommended that people using python 2.6
run with
nose, which really is no longer an option. Attila's abandoned patch
that I
linked above documents points to this bug with a patch to discover
which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that didn't
work with the PowerVM driver in Nova but I'm not aware of something
similar with testr.

Another question I guess, and I know this is too late now, but the
nosexcover package allowed us to get xunit xml results for the test runs
and then jenkins has a nice way to process that xml and keep a history
of failures and dig into the stack traces, how long the tests take,
trends, etc. I'm assuming there is nothing like that available with testr?

--

Thanks,

Matt Riedemann

responded Feb 27, 2014 by Matt_Riedemann (48,320 points)   3 11 31
0 votes

On Feb 26, 2014 7:37 PM, "Matt Riedemann"
wrote:

On 2/25/2014 7:46 PM, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with
nosetests is going
away. We've started using newer features that nose just doesn't
support. One
example of this is that we've started using testscenarios and
we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the
requirements list and all the workarounds and references to nose
will be pulled
out of the tree. Tempest will also start raising an unsupported
exception when
you try to run it with nose so that there isn't any confusion on
this moving
forward. We talked about doing this at summit briefly and I've
brought it up a
couple of times before, but I believe it is time to do this now. I
feel for
tempest to move forward we need to do this now so that there isn't
any ambiguity
as we add even more features and new types of testing.

I'm with you up to here.

Now, this will have implications for people running tempest with
python 2.6
since up until now we've set nosetests. There is a workaround for
getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported
on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it
basically has
been for while now just we've gone without making it official)

The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just
doesn't
work with tempest, and we're adding more features to tempest, your
negative test
generator being one of them, that interfere further with nose. I've
seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same
after this
change. The only thing that we'll be doing is actively preventing nose
from
working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling
nose the
only path to run tempest with py2.6 is to use testr. (which is what it
always
should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in
the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs
to be done
is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6.
(The only
difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior
while trying
to fix compatibility with nose. That's why I think we need a clear
message on
this sooner than later. Which is why I'm proposing actively raising
an error
when things are run with nose upfront so there isn't any illusion
that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
Nose support is independent of python 2.6 support. Py26 I would
still consider
to be unsupported, the issue is that the hack to make py26 work is
outside of
tempest. This is why we've recommended that people using python 2.6
run with
nose, which really is no longer an option. Attila's abandoned patch
that I
linked above documents points to this bug with a patch to discover
which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that didn't
work with the PowerVM driver in Nova but I'm not aware of something
similar with testr.

Another question I guess, and I know this is too late now, but the
nosexcover package allowed us to get xunit xml results for the test runs
and then jenkins has a nice way to process that xml and keep a history of
failures and dig into the stack traces, how long the tests take, trends,
etc. I'm assuming there is nothing like that available with testr?

--

Thanks,

Matt Riedemann


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

There is a subunit to xunit converter bundled with python-subunit.
subunit2xunit maybe? On a phone so hard to check.

Also fwiw using Jenkins to mangle that data doesn't scale well which is why
we don't do it. This may or may not be a problem. In particular the xunit
plugin has a global lock and jobs that use it can't finish until the
previous job has had its xunit processed. This is terrible.

Clark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Feb 27, 2014 by clark.boylan_at_gmai (1,500 points)   1 3
0 votes

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with nosetests is going
away. We've started using newer features that nose just doesn't support. One
example of this is that we've started using testscenarios and we're planning to
do this in more places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove nosetests from the
requirements list and all the workarounds and references to nose will be pulled
out of the tree. Tempest will also start raising an unsupported exception when
you try to run it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've brought it up a
couple of times before, but I believe it is time to do this now. I feel for
tempest to move forward we need to do this now so that there isn't any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with python 2.6
since up until now we've set nosetests. There is a workaround for getting
tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as unsupported on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.

So honestly we're already at this point in tempest. Nose really just doesn't
work with tempest, and we're adding more features to tempest, your negative test
generator being one of them, that interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same after this
change. The only thing that we'll be doing is actively preventing nose from
working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling nose the
only path to run tempest with py2.6 is to use testr. (which is what it always
should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs to be done
is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6. (The only
difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior while trying
to fix compatibility with nose. That's why I think we need a clear message on
this sooner than later. Which is why I'm proposing actively raising an error
when things are run with nose upfront so there isn't any illusion that things
are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile category.
Nose support is independent of python 2.6 support. Py26 I would still consider
to be unsupported, the issue is that the hack to make py26 work is outside of
tempest. This is why we've recommended that people using python 2.6 run with
nose, which really is no longer an option. Attila's abandoned patch that I
linked above documents points to this bug with a patch to discover which is
need to get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file
to specify the test bucket and what can be excluded, like you can
with nose.cfg and nose? We used that for filtering out API tests
that didn't work with the PowerVM driver in Nova but I'm not aware
of something similar with testr.

So I'm not sure if it's exactly what you're looking for but testr has the
--load-list option which you can use to specify a file which lists the tests
you want to run. I don't think there is a method to exclude tests besides using
a regex filter right now. There is a bug open about this:

https://bugs.launchpad.net/testrepository/+bug/1208610

So I can see you doing this 2 ways either writing a little script that will
generate a list file by doing something like:

  1. testr list-tests > file
  2. remove excludes from file
  3. testr run --load-list file

or making a long unwieldy regex that excludes the tests you need to. Something
like what I did here:

https://review.openstack.org/#/c/51275/4/tox.ini

-Matt Treinish

responded Feb 27, 2014 by Matthew_Treinish (11,200 points)   2 5 6
0 votes

This is another example of achieving the same result (exclusion from a
list):
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.py
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.txt

andrea

-----Original Message-----
From: Matthew Treinish [mailto:mtreinish at kortar.org]
Sent: 27 February 2014 15:49
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with
nosetests is going away. We've started using newer features that
nose just doesn't support. One example of this is that we've
started using testscenarios and we're planning to do this in more
places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the requirements list and all the workarounds and
references to nose will be pulled out of the tree. Tempest will
also start raising an unsupported exception when you try to run
it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've
brought it up a couple of times before, but I believe it is time
to do this now. I feel for tempest to move forward we need to do this
now so that there isn't any ambiguity as we add even more features and new
types of testing.
I'm with you up to here.

Now, this will have implications for people running tempest with
python 2.6 since up until now we've set nosetests. There is a
workaround for getting tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as
unsupported on tempest python 2.6 will also be unsupported by
Tempest. (which honestly it basically has been for while now just
we've gone without making it official)
The way we handle different runners/os can be categorized as
"tested in gate", "unsupported" (should work, possibly some hacks
needed), and "hostile". At present, both nose and py2.6 I would
say are in the unsupported category. The title of this message and
the content up to here says we are moving nose to the hostile
category. With only 2 months to feature freeze I see no
justification in moving
py2.6 to the hostile category. I don't see what new testing
features scheduled for the next two months will be enabled by
saying that tempest cannot and will not run on 2.6. It has been
agreed I think by all projects that py2.6 will be dropped in J. It
is OK that py2.6 will require some hacks to work and if in the
next few months it needs a few more then that is ok. If I am
missing another connection between the py2.6 and nose issues, please
explain.

So honestly we're already at this point in tempest. Nose really
just doesn't work with tempest, and we're adding more features to
tempest, your negative test generator being one of them, that
interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.

So as I said before the python 2.6 story for tempest remains the same
after this change. The only thing that we'll be doing is actively
preventing nose from working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.

Yes, that is what the discover patch is used to enable. By disabling
nose the only path to run tempest with py2.6 is to use testr. (which
is what it always should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in
the gate for tempest. (https://review.openstack.org/#/c/72388/ ) So
all that needs to be done is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6.
(The only difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior
while trying to fix compatibility with nose. That's why I think we
need a clear message on this sooner than later. Which is why I'm
proposing actively raising an error when things are run with nose
upfront so there isn't any illusion that things are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
Nose support is independent of python 2.6 support. Py26 I would
still consider to be unsupported, the issue is that the hack to
make py26 work is outside of tempest. This is why we've recommended
that people using python 2.6 run with nose, which really is no
longer an option. Attila's abandoned patch that I linked above
documents points to this bug with a patch to discover which is need to
get python 2.6 working with tempest and testr:

https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that
didn't work with the PowerVM driver in Nova but I'm not aware of
something similar with testr.

So I'm not sure if it's exactly what you're looking for but testr has the
--load-list option which you can use to specify a file which lists the tests
you want to run. I don't think there is a method to exclude tests besides
using a regex filter right now. There is a bug open about this:

https://bugs.launchpad.net/testrepository/+bug/1208610

So I can see you doing this 2 ways either writing a little script that will
generate a list file by doing something like:

  1. testr list-tests > file
  2. remove excludes from file
  3. testr run --load-list file

or making a long unwieldy regex that excludes the tests you need to.
Something like what I did here:

https://review.openstack.org/#/c/51275/4/tox.ini

-Matt Treinish


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6199 bytes
Desc: not available
URL:
responded Feb 27, 2014 by Frittoli,_Andrea_(HP (840 points)   2
0 votes

Hi,

Let me express my concerns on this topic:

With some recent changes made to Tempest compatibility with
nosetests is going away.
I think that we should not drop nosetests support from tempest or any
other project. The problem with testrepository is that it's not
providing any debugger support at all (and will never provide). It also
has some issues with proving error traces in human readable form and it
can be quite hard to find out what is actually broken.

Because of this I think we should try to avoid any kind of test
libraries that break compatibility with conventional test runners.

Our tests should be able to run correctly with nosetests, teststools or
plain old unittest runner. If for some reason test libraries (like
testscenarios) doesn't provide support for this we should fix this
libraries or avoid their usage.

Regards,
Alexei Kornienko

On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:
This is another example of achieving the same result (exclusion from a
list):
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.py
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.txt

andrea

-----Original Message-----
From: Matthew Treinish [mailto:mtreinish at kortar.org]
Sent: 27 February 2014 15:49
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with
nosetests is going away. We've started using newer features that
nose just doesn't support. One example of this is that we've
started using testscenarios and we're planning to do this in more
places moving forward.
So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the requirements list and all the workarounds and
references to nose will be pulled out of the tree. Tempest will
also start raising an unsupported exception when you try to run
it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've
brought it up a couple of times before, but I believe it is time
to do this now. I feel for tempest to move forward we need to do this
now so that there isn't any ambiguity as we add even more features and new
types of testing.
I'm with you up to here.
Now, this will have implications for people running tempest with
python 2.6 since up until now we've set nosetests. There is a
workaround for getting tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as
unsupported on tempest python 2.6 will also be unsupported by
Tempest. (which honestly it basically has been for while now just
we've gone without making it official)
The way we handle different runners/os can be categorized as
"tested in gate", "unsupported" (should work, possibly some hacks
needed), and "hostile". At present, both nose and py2.6 I would
say are in the unsupported category. The title of this message and
the content up to here says we are moving nose to the hostile
category. With only 2 months to feature freeze I see no
justification in moving
py2.6 to the hostile category. I don't see what new testing
features scheduled for the next two months will be enabled by
saying that tempest cannot and will not run on 2.6. It has been
agreed I think by all projects that py2.6 will be dropped in J. It
is OK that py2.6 will require some hacks to work and if in the
next few months it needs a few more then that is ok. If I am
missing another connection between the py2.6 and nose issues, please
explain.
So honestly we're already at this point in tempest. Nose really
just doesn't work with tempest, and we're adding more features to
tempest, your negative test generator being one of them, that
interfere further with nose. I've seen several
I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).

I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.

If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.
So as I said before the python 2.6 story for tempest remains the same
after this change. The only thing that we'll be doing is actively
preventing nose from working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.

What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.
Yes, that is what the discover patch is used to enable. By disabling
nose the only path to run tempest with py2.6 is to use testr. (which
is what it always should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in
the gate for tempest. (https://review.openstack.org/#/c/72388/ ) So
all that needs to be done is to apply that discover patch:

https://code.google.com/p/unittest-ext/issues/detail?id=79

(which I linked to before)

Then tempest should run more or less the same between 2.7 and 2.6.
(The only difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect behavior
while trying to fix compatibility with nose. That's why I think we
need a clear message on this sooner than later. Which is why I'm
proposing actively raising an error when things are run with nose
upfront so there isn't any illusion that things are expected to work.

This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
Nose support is independent of python 2.6 support. Py26 I would
still consider to be unsupported, the issue is that the hack to
make py26 work is outside of tempest. This is why we've recommended
that people using python 2.6 run with nose, which really is no
longer an option. Attila's abandoned patch that I linked above
documents points to this bug with a patch to discover which is need to
get python 2.6 working with tempest and testr:
https://code.google.com/p/unittest-ext/issues/detail?id=79


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that
didn't work with the PowerVM driver in Nova but I'm not aware of
something similar with testr.
So I'm not sure if it's exactly what you're looking for but testr has the
--load-list option which you can use to specify a file which lists the tests
you want to run. I don't think there is a method to exclude tests besides
using a regex filter right now. There is a bug open about this:

https://bugs.launchpad.net/testrepository/+bug/1208610

So I can see you doing this 2 ways either writing a little script that will
generate a list file by doing something like:

  1. testr list-tests > file
  2. remove excludes from file
  3. testr run --load-list file

or making a long unwieldy regex that excludes the tests you need to.
Something like what I did here:

https://review.openstack.org/#/c/51275/4/tox.ini

-Matt Treinish


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Feb 28, 2014 by Alexei_Kornienko (1,060 points)   1 1
0 votes

----- Original Message -----

From: "Alexei Kornienko" <alexei.kornienko at gmail.com>
To: openstack-dev at lists.openstack.org
Sent: Friday, February 28, 2014 12:43:13 PM
Subject: Re: [openstack-dev] [QA] The future of nosetests with
Tempest

Hi,

Let me express my concerns on this topic:

With some recent changes made to Tempest compatibility with

nosetests is going away.

I think that we should not drop nosetests support from tempest or any
other project. The problem with testrepository is that it's not
providing any debugger support at all (and will never provide). It
also has some issues with proving error traces in human readable
form and it can be quite hard to find out what is actually broken.

Because of this I think we should try to avoid any kind of test
libraries that break compatibility with conventional test runners.

Our tests should be able to run correctly with nosetests, teststools
or plain old unittest runner. If for some reason test libraries
(like testscenarios) doesn't provide support for this we should fix
this libraries or avoid their usage.

+1
I have the same concern about debugging. It's an essential tool (for me, at least) in creating scenario tests. The more complex the test, the harder it is to rely on simple log-prints.

Regards,
Alexei Kornienko

On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:

This is another example of achieving the same result (exclusion
from
a

list):
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.py
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.txt

andrea

-----Original Message-----

From: Matthew Treinish [ mailto:mtreinish at kortar.org ]

Sent: 27 February 2014 15:49

To: OpenStack Development Mailing List (not for usage questions)

Subject: Re: [openstack-dev] [QA] The future of nosetests with
Tempest

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:

On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:

On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz
wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility
with

nosetests is going away. We've started using newer
features
that

nose just doesn't support. One example of this is that
we've

started using testscenarios and we're planning to do
this
in
more

places moving forward.

So at Icehouse-3 I'm planning to push the patch out to
remove

nosetests from the requirements list and all the
workarounds
and

references to nose will be pulled out of the tree.
Tempest
will

also start raising an unsupported exception when you
try
to
run

it with nose so that there isn't any confusion on this
moving

forward. We talked about doing this at summit briefly
and
I've

brought it up a couple of times before, but I believe
it
is
time

to do this now. I feel for tempest to move forward we
need
to
do
this

now so that there isn't any ambiguity as we add even more features
and new

types of testing.

I'm with you up to here.

Now, this will have implications for people running
tempest
with

python 2.6 since up until now we've set nosetests.
There
is
a

workaround for getting tempest to run with python 2.6
and
testr
see:
https://review.openstack.org/#/c/59007/1/README.rst but
essentially
this means that when nose is marked as

unsupported on tempest python 2.6 will also be
unsupported
by

Tempest. (which honestly it basically has been for
while
now
just

we've gone without making it official)

The way we handle different runners/os can be categorized
as

"tested in gate", "unsupported" (should work, possibly
some
hacks

needed), and "hostile". At present, both nose and py2.6 I
would

say are in the unsupported category. The title of this
message
and

the content up to here says we are moving nose to the
hostile

category. With only 2 months to feature freeze I see no

justification in moving

py2.6 to the hostile category. I don't see what new
testing

features scheduled for the next two months will be
enabled
by

saying that tempest cannot and will not run on 2.6. It
has
been

agreed I think by all projects that py2.6 will be dropped
in
J.
It

is OK that py2.6 will require some hacks to work and if
in
the

next few months it needs a few more then that is ok. If I
am

missing another connection between the py2.6 and nose
issues,
please

explain.

So honestly we're already at this point in tempest. Nose
really

just doesn't work with tempest, and we're adding more
features
to

tempest, your negative test generator being one of them,
that

interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and
scenario

tests every day with nose on RHEL 6 with minimal issues. I
had
to

workaround the negative test discovery by simply sed'ing that
out
of

the tests before running it, but that's acceptable to me
until
we

can start testing on RHEL 7. Otherwise I'm completely OK
with

saying py26 isn't really supported and isn't used in the
gate,
and

it's a buyer beware situation to make it work, which includes

pushing up trivial patches to make it work (which I did a few
of

last week, and they were small syntax changes or usages of

testtools).

I don't understand how the core projects can be running unit
tests

in the gate on py26 but our functional integration project is
going

to actively go out and make it harder to run Tempest with
py26,
that

sucks.

If we really want to move the test project away from py26,
let's

make the concerted effort to get the core projects to move
with
it.

So as I said before the python 2.6 story for tempest remains
the
same

after this change. The only thing that we'll be doing is
actively

preventing nose from working with tempest.

And FWIW, I tried the discover.py patch with unittest2 and

testscenarios last week and either I botched it, it's not
documented

properly on how to apply it, or I screwed something up, but
it

didn't work for me, so I'm not convinced that's the
workaround.

What's the other option for running Tempest on py26 (keeping
RHEL
6

in mind)? Using tox with testr and pip? I'm doing this all

single-node.

Yes, that is what the discover patch is used to enable. By
disabling

nose the only path to run tempest with py2.6 is to use testr.
(which

is what it always should have been)

Attila confirmed it was working here:
http://fpaste.org/76651/32143139/ in that example he applies 2
patches the second one is currently in

the gate for tempest. ( https://review.openstack.org/#/c/72388/
)
So

all that needs to be done is to apply that discover patch:
https://code.google.com/p/unittest-ext/issues/detail?id=79
(which
I
linked to before)

Then tempest should run more or less the same between 2.7 and
2.6.

(The only difference I've seen is in how skips are handled)

patches this cycle that attempted to introduce incorrect
behavior

while trying to fix compatibility with nose. That's why I
think
we

need a clear message on this sooner than later. Which is
why
I'm

proposing actively raising an error when things are run
with
nose

upfront so there isn't any illusion that things are
expected
to
work.

This doesn't necessarily mean we're moving python 2.6 to
the
hostile

category.

Nose support is independent of python 2.6 support. Py26 I
would

still consider to be unsupported, the issue is that the
hack
to

make py26 work is outside of tempest. This is why we've
recommended

that people using python 2.6 run with nose, which really is
no

longer an option. Attila's abandoned patch that I linked
above

documents points to this bug with a patch to discover which
is
need
to

get python 2.6 working with tempest and testr:

One question I had was is there an easy way to setup a config
file
to

specify the test bucket and what can be excluded, like you can
with

nose.cfg and nose? We used that for filtering out API tests that

didn't work with the PowerVM driver in Nova but I'm not aware of

something similar with testr.

So I'm not sure if it's exactly what you're looking for but testr
has
the

--load-list option which you can use to specify a file which lists
the tests

you want to run. I don't think there is a method to exclude tests
besides

using a regex filter right now. There is a bug open about this:
https://bugs.launchpad.net/testrepository/+bug/1208610 So I can see
you doing this 2 ways either writing a little script that will

generate a list file by doing something like:

  1. testr list-tests > file

  2. remove excludes from file

  3. testr run --load-list file

or making a long unwieldy regex that excludes the tests you need
to.

Something like what I did here:
https://review.openstack.org/#/c/51275/4/tox.ini -Matt Treinish


OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 6, 2014 by Yair_Fried (800 points)   1
retagged Jan 28, 2015 by admin
...