settingsLogin | Registersettings

[openstack-dev] How to turn tempest CLI tests into python-*client in-tree functional tests

0 votes

1.
A few months back we started the process to remove the tempest CLI
tests from tempest [0]. Now that we have successfully pulled
novaclient CLI
tests out of tempest, we have the process sorted out. We now
have a process
that should be easy to follow for each project, in fact
keystoneclient has
already begun as well [1]. As stated in [0], the goal is to completely
remove CLI tests from tempest by the end of the cycle.

  [0]
  http://lists.openstack.org/pipermail/openstack-dev/2014-October/048089.html

[1] https://review.openstack.org/#/c/155543/

  *Steps*
asked Feb 13, 2015 in openstack-dev by Joe_Gordon (24,620 points)   2 5 8
retagged Mar 6, 2015 by admin

6 Responses

0 votes

What's the test path thing for? Testr should be able to filter out unit
tests or vice versa without altering discovery.
On 14 Feb 2015 08:57, "Joe Gordon" joe.gordon0@gmail.com wrote:

1.
A few months back we started the process to remove the tempest CLI
tests from tempest [0]. Now that we have successfully pulled novaclient CLI
tests out of tempest, we have the process sorted out. We now have a process
that should be easy to follow for each project, in fact keystoneclient has
already begun as well [1]. As stated in [0], the goal is to completely
remove CLI tests from tempest by the end of the cycle.

  [0]
  http://lists.openstack.org/pipermail/openstack-dev/2014-October/048089.html

[1] https://review.openstack.org/#/c/155543/

  *Steps*


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 13, 2015 by Robert_Collins (27,200 points)   4 6 11
0 votes

Digging through the logs this originated from this bug:
https://bugs.launchpad.net/tempest/+bug/1260710

Its probably not needed everywhere and in all the clients.

On Fri, Feb 13, 2015 at 1:06 PM, Robert Collins robertc@robertcollins.net
wrote:

What's the test path thing for? Testr should be able to filter out unit
tests or vice versa without altering discovery.
On 14 Feb 2015 08:57, "Joe Gordon" joe.gordon0@gmail.com wrote:

1.
A few months back we started the process to remove the tempest CLI
tests from tempest [0]. Now that we have successfully pulled novaclient CLI
tests out of tempest, we have the process sorted out. We now have a process
that should be easy to follow for each project, in fact keystoneclient has
already begun as well [1]. As stated in [0], the goal is to completely
remove CLI tests from tempest by the end of the cycle.

  [0]
  http://lists.openstack.org/pipermail/openstack-dev/2014-October/048089.html

[1] https://review.openstack.org/#/c/155543/

  *Steps*


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 13, 2015 by Joe_Gordon (24,620 points)   2 5 8
0 votes

On Fri, Feb 13, 2015 at 11:57 AM, Joe Gordon joe.gordon0@gmail.com wrote:

1.
A few months back we started the process to remove the tempest CLI
tests from tempest [0]. Now that we have successfully pulled novaclient CLI
tests out of tempest, we have the process sorted out. We now have a process
that should be easy to follow for each project, in fact keystoneclient has
already begun as well [1]. As stated in [0], the goal is to completely
remove CLI tests from tempest by the end of the cycle.

  [0]
  http://lists.openstack.org/pipermail/openstack-dev/2014-October/048089.html

[1] https://review.openstack.org/#/c/155543/

  *Steps*

This patch had a bug that was fixed here:
https://review.openstack.org/#/c/157845/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 20, 2015 by Joe_Gordon (24,620 points)   2 5 8
0 votes

On 14 February 2015 at 10:26, Joe Gordon joe.gordon0@gmail.com wrote:
Digging through the logs this originated from this bug:
https://bugs.launchpad.net/tempest/+bug/1260710

Its probably not needed everywhere and in all the clients.

So I've looked more closely at this.

Its actually an antipattern. It tells testr that tests are appearing
and disappearing depending on what test entry point a user runs each
time.

testr expects the set of tests to only change when code changes.

So, I fully expect that this pattern is going to lead to wtf moments
now, and likely more in future.

Whats the right forum for discussing the pressures that lead to this
hack, so we can do something that works better with the underlying
tooling, rather than in such a disruptive fashion?

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 6, 2015 by Robert_Collins (27,200 points)   4 6 11
0 votes

On Wed, 6 May 2015, Robert Collins wrote:

Its actually an antipattern. It tells testr that tests are appearing
and disappearing depending on what test entry point a user runs each
time.

testr expects the set of tests to only change when code changes.

So, I fully expect that this pattern is going to lead to wtf moments
now, and likely more in future.

Whats the right forum for discussing the pressures that lead to this
hack, so we can do something that works better with the underlying
tooling, rather than in such a disruptive fashion?

I'd appreciate here (that is on this list) because from my
perspective there are a lot of embedded assumptions in the way testr
does things and wants the environment to be that aren't immediately
obvious and would perhaps be made more clear if you could expand on
the details of what's wrong with this particular hack.

I tried to come up with a specific question here to drive that
illumination a bit more concretely but a) not enough coffee yet b)
mostly I just want to know more detail about the first three
paragraphs above.

Thanks.
--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 6, 2015 by Chris_Dent (7,940 points)   1 4 6
0 votes

On 05/06/2015 04:57 AM, Chris Dent wrote:
On Wed, 6 May 2015, Robert Collins wrote:

Its actually an antipattern. It tells testr that tests are appearing
and disappearing depending on what test entry point a user runs each
time.

testr expects the set of tests to only change when code changes.

So, I fully expect that this pattern is going to lead to wtf moments
now, and likely more in future.

Whats the right forum for discussing the pressures that lead to this
hack, so we can do something that works better with the underlying
tooling, rather than in such a disruptive fashion?

I'd appreciate here (that is on this list) because from my
perspective there are a lot of embedded assumptions in the way testr
does things and wants the environment to be that aren't immediately
obvious and would perhaps be made more clear if you could expand on
the details of what's wrong with this particular hack.

I tried to come up with a specific question here to drive that
illumination a bit more concretely but a) not enough coffee yet b)
mostly I just want to know more detail about the first three
paragraphs above.

There are 2 reasons that pattern exists.

1) testr discovery is quite slow on large trees, especially if your
intent is to run a small subset of tests by sending an argument

2) testr still doesn't have an exclude facility, so top level test
exclusion has to be done by quite complicated negative asserting regex,
which are very error prone (and have been done incorrectly many times).
Especially if you still want to support partial test passing.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 6, 2015 by Sean_Dague (66,200 points)   4 8 14
...