settingsLogin | Registersettings

[openstack-dev] [Neutron][QA] Call to action - Neutron/Tempest API tests dedup

0 votes

TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
Help would be hugely appreciated and %(local_drink)s will be bought in
Barcelona. I've posted example patches that demonstrate the idea.
Needless to say I'm here to provide reviews and to answer questions.
Additionally, most of the discussions have been within the Neutron
community and I'm looking for feedback from Tempest folks.

The context:
The Neutron community has been engaged in a long running effort to
move some of the networking API tests to the Neutron tree. We started
by copying the api/network directory tree, later keeping only the
tests, importing the test infrastructure itself from Tempest. We
continued by minimizing the imports we do from Tempest (Excluding
tempest.lib), and introduced a Neutron Tempest plugin.

One issue that remains is that some of the tests are still found in
both repositories. This confuses contributors and wastes compute
resources. Since the tests run against stable/{liberty|mitaka} and
master, it should be safe to dedup. I proposed a line in the sand so
that 'core resources' remain to be tested in Tempest and more
'advanced' APIs are tested in Neutron. The concept was agreed upon by
the Neutron and (Then) Tempest PTLs, and the specifics were discussed
and a consensus was found in patch [2]. Here is the resulting doc for
your viewing pleasure [5].

The work:
After I removed the API tests for core resources from the Neutron
tree, there remain three tasks to finish the de-dup:

1*) Remove tests for advanced APIs from Tempest. The full list of
tests that I propose be removed from Tempest is tracked here [1] (With
the rationale found at [2]), and an example patch may be found here
[4].
2) Push tests for Neutron core resources that were added after the
fork from Tempest, then delete these from Neutron. This is also
tracked in [1], with example patches found here [6]. This is not a
strict cut/paste as the way Tempest and Neutron interact with clients
is slightly different. Fun!
3) Sync tests for Neutron core resources that were updated after the
fork from Tempest. Test modifications include: Bug fixes for raceful
tests, py3 fixes, doc string typos and more. This is also tracked in
[1], with example patches found here [3].

  • I believe that as far as the Tempest test removal criteria found at
    [7], this case falls under the first exception: 'The class of testing
    has been decided to be outside the scope of tempest' and we may skip
    the three prong rule for removal. Input welcome.

[1] https://etherpad.openstack.org/p/neutron-tempest-defork
[2] https://review.openstack.org/#/c/280427/
[3] https://review.openstack.org/#/c/316280/ +
https://review.openstack.org/#/c/316283/
[4] https://review.openstack.org/#/c/316183/
[5] docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
[6] https://review.openstack.org/#/c/316265/ +
https://review.openstack.org/#/c/316269/
[7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal

The work is tracked via:
* https://review.openstack.org/#/q/topic:bug/1552960
* https://bugs.launchpad.net/neutron/+bug/1552960
* https://etherpad.openstack.org/p/neutron-tempest-defork
* My head


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 13, 2016 in openstack-dev by assaf_at_redhat.com (2,500 points)   1 1
retagged Jan 25, 2017 by admin

2 Responses

0 votes

hi, Muller.

As you mentioned, is there will have a individual Tempest plugin

named "Neutron
Tempest plugin" in future?

Thanks
Zhi Chang

2016-05-14 6:53 GMT+08:00 Assaf Muller assaf@redhat.com:

TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
Help would be hugely appreciated and %(local_drink)s will be bought in
Barcelona. I've posted example patches that demonstrate the idea.
Needless to say I'm here to provide reviews and to answer questions.
Additionally, most of the discussions have been within the Neutron
community and I'm looking for feedback from Tempest folks.

The context:
The Neutron community has been engaged in a long running effort to
move some of the networking API tests to the Neutron tree. We started
by copying the api/network directory tree, later keeping only the
tests, importing the test infrastructure itself from Tempest. We
continued by minimizing the imports we do from Tempest (Excluding
tempest.lib), and introduced a Neutron Tempest plugin.

One issue that remains is that some of the tests are still found in
both repositories. This confuses contributors and wastes compute
resources. Since the tests run against stable/{liberty|mitaka} and
master, it should be safe to dedup. I proposed a line in the sand so
that 'core resources' remain to be tested in Tempest and more
'advanced' APIs are tested in Neutron. The concept was agreed upon by
the Neutron and (Then) Tempest PTLs, and the specifics were discussed
and a consensus was found in patch [2]. Here is the resulting doc for
your viewing pleasure [5].

The work:
After I removed the API tests for core resources from the Neutron
tree, there remain three tasks to finish the de-dup:

1*) Remove tests for advanced APIs from Tempest. The full list of
tests that I propose be removed from Tempest is tracked here [1] (With
the rationale found at [2]), and an example patch may be found here
[4].
2) Push tests for Neutron core resources that were added after the
fork from Tempest, then delete these from Neutron. This is also
tracked in [1], with example patches found here [6]. This is not a
strict cut/paste as the way Tempest and Neutron interact with clients
is slightly different. Fun!
3) Sync tests for Neutron core resources that were updated after the
fork from Tempest. Test modifications include: Bug fixes for raceful
tests, py3 fixes, doc string typos and more. This is also tracked in
[1], with example patches found here [3].

  • I believe that as far as the Tempest test removal criteria found at
    [7], this case falls under the first exception: 'The class of testing
    has been decided to be outside the scope of tempest' and we may skip
    the three prong rule for removal. Input welcome.

[1] https://etherpad.openstack.org/p/neutron-tempest-defork
[2] https://review.openstack.org/#/c/280427/
[3] https://review.openstack.org/#/c/316280/ +
https://review.openstack.org/#/c/316283/
[4] https://review.openstack.org/#/c/316183/
[5]
docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
[6] https://review.openstack.org/#/c/316265/ +
https://review.openstack.org/#/c/316269/
[7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal

The work is tracked via:
* https://review.openstack.org/#/q/topic:bug/1552960
* https://bugs.launchpad.net/neutron/+bug/1552960
* https://etherpad.openstack.org/p/neutron-tempest-defork
* My head


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 14, 2016 by =?UTF-8?B?6IuM5pm6?= (1,680 points)   3 9 18
0 votes

On Sat, May 14, 2016 at 8:03 AM, zhi changzhi1990@gmail.com wrote:
hi, Muller.

As you mentioned, is there will have a individual Tempest plugin named

"Neutron Tempest plugin" in future?

It already exists :)

http://docs.openstack.org/developer/neutron/devref/development.environment.html#id3

The plugin was contributed by Daniel Mellado and we use it at the gate:
https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/neutron.yaml#L34

Thanks
Zhi Chang

2016-05-14 6:53 GMT+08:00 Assaf Muller assaf@redhat.com:

TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
Help would be hugely appreciated and %(local_drink)s will be bought in
Barcelona. I've posted example patches that demonstrate the idea.
Needless to say I'm here to provide reviews and to answer questions.
Additionally, most of the discussions have been within the Neutron
community and I'm looking for feedback from Tempest folks.

The context:
The Neutron community has been engaged in a long running effort to
move some of the networking API tests to the Neutron tree. We started
by copying the api/network directory tree, later keeping only the
tests, importing the test infrastructure itself from Tempest. We
continued by minimizing the imports we do from Tempest (Excluding
tempest.lib), and introduced a Neutron Tempest plugin.

One issue that remains is that some of the tests are still found in
both repositories. This confuses contributors and wastes compute
resources. Since the tests run against stable/{liberty|mitaka} and
master, it should be safe to dedup. I proposed a line in the sand so
that 'core resources' remain to be tested in Tempest and more
'advanced' APIs are tested in Neutron. The concept was agreed upon by
the Neutron and (Then) Tempest PTLs, and the specifics were discussed
and a consensus was found in patch [2]. Here is the resulting doc for
your viewing pleasure [5].

The work:
After I removed the API tests for core resources from the Neutron
tree, there remain three tasks to finish the de-dup:

1*) Remove tests for advanced APIs from Tempest. The full list of
tests that I propose be removed from Tempest is tracked here [1] (With
the rationale found at [2]), and an example patch may be found here
[4].
2) Push tests for Neutron core resources that were added after the
fork from Tempest, then delete these from Neutron. This is also
tracked in [1], with example patches found here [6]. This is not a
strict cut/paste as the way Tempest and Neutron interact with clients
is slightly different. Fun!
3) Sync tests for Neutron core resources that were updated after the
fork from Tempest. Test modifications include: Bug fixes for raceful
tests, py3 fixes, doc string typos and more. This is also tracked in
[1], with example patches found here [3].

  • I believe that as far as the Tempest test removal criteria found at
    [7], this case falls under the first exception: 'The class of testing
    has been decided to be outside the scope of tempest' and we may skip
    the three prong rule for removal. Input welcome.

[1] https://etherpad.openstack.org/p/neutron-tempest-defork
[2] https://review.openstack.org/#/c/280427/
[3] https://review.openstack.org/#/c/316280/ +
https://review.openstack.org/#/c/316283/
[4] https://review.openstack.org/#/c/316183/
[5]
docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
[6] https://review.openstack.org/#/c/316265/ +
https://review.openstack.org/#/c/316269/
[7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal

The work is tracked via:
* https://review.openstack.org/#/q/topic:bug/1552960
* https://bugs.launchpad.net/neutron/+bug/1552960
* https://etherpad.openstack.org/p/neutron-tempest-defork
* My head


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 14, 2016 by assaf_at_redhat.com (2,500 points)   1 1
...