settingsLogin | Registersettings

[openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

0 votes

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client for
the Heat project.
The Heat service client is used by the tests in Tempest, which run in Heat
gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the layer4
job.
According to code search [3] the Heat service client is also used by Murano
and Daisycore.

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that all
tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

It would propose to move tests and client to a Tempest plugin owned /
maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to use
the Heat service
client in their tests, even if the client is removed from Tempest, it would
still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

Andrea Frittoli (andreaf)

[0]
https://docs.openstack.org/developer/tempest/test_removal.html#tempest-scope
[1] https://docs.openstack.org/developer/tempest/plugin.html
[2] https://docs.openstack.org/developer/tempest/library.html
[3]
http://codesearch.openstack.org/?q=self.orchestration_client&i=nope&files=&repos=

[4] https://review.openstack.org/#/c/456843/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 4, 2017 in openstack-dev by Andrea_Frittoli (5,920 points)   1 2 3

13 Responses

0 votes

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client for
the Heat project.
The Heat service client is used by the tests in Tempest, which run in Heat
gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the layer4
job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade' phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that all
tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to use
the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

Andrea Frittoli (andreaf)

--
Regards,
Rabi Mishra


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 28, 2017 by Rabi_Mishra (2,140 points)   3 8
0 votes

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade' phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and service
client out of Tempest.

andrea

[5] https://review.openstack.org/#/c/369749/

Andrea Frittoli (andreaf)

--
Regards,
Rabi Mishra


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 28, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we do
in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /
maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and service
client out of Tempest.

andrea

Other than grenade and layer4 job on Tempest, do we run heat tests
anywhere else? With Rabi's proposed changes on grenade, If tempest
tests are not going to run on gate any more and heat team is going to
improve the heat in-tree integration tests coverage, then we can just
remove tests and corresponding config options as Andrea proposed.

Otherwise, IMO it should be done with separate repo to avoid more work
once proposed TC goal is approved. But we should make decision on
tempest heat tests now as its been long time those are maintained in
Tempest.

[5] https://review.openstack.org/#/c/369749/

--
Regards,
Rabi Mishra


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 30, 2017 by GHANSHYAM_MANN (5,700 points)   1 3 4
0 votes

Murano currently uses the Tempest orchestration client for its scenario Tempest tests [0], which are not turned on by default in the Murano Tempest gate due to resource constraints.

However, I'm hesitant to switch to Heat's testing client, because it is not a Tempest client, but rather the python-heatclient. I would like to know whether there are plans to change this to a Tempest-based client?

[0] https://github.com/openstack/murano/blob/master/murano_tempest_tests/tests/scenario/application_catalog/base.py#L100
[1] https://github.com/openstack/heat/blob/master/heat_integrationtests/common/clients.py#L120

Felipe

-----Original Message-----
From: Ghanshyam Mann [mailto:ghanshyammann@gmail.com]
Sent: Sunday, April 30, 2017 1:53 AM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_460542_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=d2pZwZ8xKsFLHxQ0YNiM4itJjUHzgE0ibHNu7v28mXM&e=

  1. To remove tempest tests from the grenade job

https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_460810_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=07__zljUdvdtD_K5ltoKwdjaBwrs0fYJKaXSr93AAiU&e=

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we do
in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /
maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and service
client out of Tempest.

andrea

Other than grenade and layer4 job on Tempest, do we run heat tests
anywhere else? With Rabi's proposed changes on grenade, If tempest
tests are not going to run on gate any more and heat team is going to
improve the heat in-tree integration tests coverage, then we can just
remove tests and corresponding config options as Andrea proposed.

Otherwise, IMO it should be done with separate repo to avoid more work
once proposed TC goal is approved. But we should make decision on
tempest heat tests now as its been long time those are maintained in
Tempest.

[5] https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_369749_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=8C3kPSTqpT4tFcIUEC1bYVUgAPC-W08UECtKEAck6mc&e=

Andrea Frittoli (andreaf)

[0]
https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_developer_tempest_test-5Fremoval.html-23tempest-2Dscope&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=zR8OxN2YWCkTSqwOdXO86ijrTvhfcrz-jTcYDWcerSI&e=
[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_developer_tempest_plugin.html&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=JTzZ4W8taIX9LY35e5yEZLFpBJT0NfHlkQ9Fdm875lE&e=
[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_developer_tempest_library.html&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=b4aI4XWUkF8-yJy_EGtTpAJ9O47LQSFzz6oplawJuE4&e=
[3]
https://urldefense.proofpoint.com/v2/url?u=http-3A__codesearch.openstack.org_-3Fq-3Dself.orchestration-5Fclient-26i-3Dnope-26files-3D-26repos-3D&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=Lw4Nep7Zyf-sUfbKo8DeOjVzVMWKxOwfD3G2MzZXjAQ&e=
[4] https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_456843_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=MkpWWcQzrowgdxFsD1P4Mn4XvEKvM2Ym8DMKcq5KLNg&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF_OPLkwsINaR4MckrDmIQ2drs&e=

--
Regards,
Rabi Mishra


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF_OPLkwsINaR4MckrDmIQ2drs&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF_OPLkwsINaR4MckrDmIQ2drs&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF_OPLkwsINaR4MckrDmIQ2drs&e=

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 1, 2017 by MONTEIRO,_FELIPE_C (520 points)  
0 votes

On Tue, May 2, 2017 at 5:27 AM, MONTEIRO, FELIPE C fm577c@att.com wrote:

Murano currently uses the Tempest orchestration client for its scenario
Tempest tests [0], which are not turned on by default in the Murano Tempest
gate due to resource constraints.

However, I'm hesitant to switch to Heat's testing client, because it is
not a Tempest client, but rather the python-heatclient. I would like to
know whether there are plans to change this to a Tempest-based client?

There are no plans to switch the heat integration/functional tests to using
the tempest based client. The heat tests will use heatclient for most
tests, and gabbi for testing the REST API.

Since you're testing Murano rather than the Heat API, I think converting
your tests to heatclient would be reasonable.

[0] https://github.com/openstack/murano/blob/master/murano_
tempesttests/tests/scenario/applicationcatalog/base.py#L100
[1] https://github.com/openstack/heat/blob/master/heat_
integrationtests/common/clients.py#L120

Felipe

-----Original Message-----
From: Ghanshyam Mann [mailto:ghanshyammann@gmail.com]
Sent: Sunday, April 30, 2017 1:53 AM
To: OpenStack Development Mailing List (not for usage questions) <
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat
support from Tempest

On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com
wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
andrea.frittoli@gmail.com wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects
which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova,
glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
openstack.org-23c460542&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-
SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=
d2pZwZ8xKsFLHxQ0YNiM4itJjUHzgE0ibHNu7v28mXM&e=

  1. To remove tempest tests from the grenade job

https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
openstack.org-23c460810&d=DwIGaQ&c=LFYZ-o9HUMeMTSQicvjIg&r=X4GwEru-
SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=07__
zljUdvdtD
K5ltoKwdjaBwrs0fYJKaXSr93AAiU&e=

I proposed a patch to Tempest to start the deprecation counter for
Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end
the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I
don't
know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the
gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as
we do
in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /
maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service
client
manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team.
Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing
pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to
a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion
about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and
service
client out of Tempest.

andrea

Other than grenade and layer4 job on Tempest, do we run heat tests
anywhere else? With Rabi's proposed changes on grenade, If tempest
tests are not going to run on gate any more and heat team is going to
improve the heat in-tree integration tests coverage, then we can just
remove tests and corresponding config options as Andrea proposed.

Otherwise, IMO it should be done with separate repo to avoid more work
once proposed TC goal is approved. But we should make decision on
tempest heat tests now as its been long time those are maintained in
Tempest.

[5] https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
openstack.org-23c369749&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-
SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=
8C3kPSTqpT4tFcIUEC1bYVUgAPC-W08UECtKEAck6mc&e=

Andrea Frittoli (andreaf)

[0]
https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.
openstack.orgdevelopertempesttest-5Fremoval.html-
23tempest-2Dscope&d=DwIGaQ&c=LFYZ-o9
HUMeMTSQicvjIg&r=
X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=
zR8OxN2YWCkTSqwOdXO86ijrTvhfcrz-jTcYDWcerSI&e=
[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.
openstack.orgdevelopertempestplugin.html&d=DwIGaQ&
c=LFYZ-o9
HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=JTzZ4W8taIX9LY35e5yEZLFpBJT0Nf
HlkQ9Fdm875lE&e=
[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.
openstack.orgdevelopertempestlibrary.html&d=DwIGaQ&
c=LFYZ-o9
HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=b4aI4XWUkF8-yJy_
EGtTpAJ9O47LQSFzz6oplawJuE4&e=
[3]
https://urldefense.proofpoint.com/v2/url?u=http-3A__
codesearch.openstack.org-3Fq-3Dself.orchestration-5Fclient-
26i-3Dnope-26files-3D-26repos-3D&d=DwIGaQ&c=LFYZ-o9

HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-
NeCWHJdSvrVsYA&s=Lw4Nep7Zyf-sUfbKo8DeOjVzVMWKxOwfD3G2MzZXjAQ&e=
[4] https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
openstack.org-23c456843&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=X4GwEru-
SJ9DRnCxhze-aw&m=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=
MkpWWcQzrowgdxFsD1P4Mn4XvEKvM2Ym8DMKcq5KLNg&e=



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
openstack.orgcgi-2Dbinmailmanlistinfoopenstack-
2Ddev&d=DwIGaQ&c=LFYZ-o9HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF

OPLkwsINaR4MckrDmIQ2drs&e=

--
Regards,
Rabi Mishra



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
openstack.orgcgi-2Dbinmailmanlistinfoopenstack-
2Ddev&d=DwIGaQ&c=LFYZ-o9HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF

OPLkwsINaR4MckrDmIQ2drs&e=



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
openstack.orgcgi-2Dbinmailmanlistinfoopenstack-
2Ddev&d=DwIGaQ&c=LFYZ-o9HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF

OPLkwsINaR4MckrDmIQ2drs&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
openstack.orgcgi-2Dbinmailmanlistinfoopenstack-
2Ddev&d=DwIGaQ&c=LFYZ-o9HUMeMTSQicvjIg&r=X4GwEru-SJ9DRnCxhze-aw&m=aN-
OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA&s=cjW0GYqZtCKw-P5ikZF

OPLkwsINaR4MckrDmIQ2drs&e=

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 1, 2017 by Steve_Baker (7,380 points)   1 3 7
0 votes

On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects
which are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova,
glance, swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for Heat
/ orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end
the Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I
don't know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service
client manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and
service client out of Tempest.

We probably are mixing two different things here.

  1. Moving in-tree heat templest plugn and tests to a dedicated repo

Though we don't have any plans for it now, we may have to do it when/if
it's accepted as a community goal.

  1. Moving tempest tree heat tests and heat service client to a new home
    and owner.

I don't think that's something heat team would like to do given that we
don't use these tests anywhere and would probably spend time improving the
coverage of the gabbi api tests we already have.

andrea

[5] https://review.openstack.org/#/c/369749/

Andrea Frittoli (andreaf)

--
Regards,
Rabi Mishra



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Rabi Mishra


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 2, 2017 by Rabi_Mishra (2,140 points)   3 8
0 votes

----- Original Message -----

On Tue, May 2, 2017 at 5:27 AM, MONTEIRO, FELIPE C < fm577c@att.com > wrote:

Murano currently uses the Tempest orchestration client for its scenario
Tempest tests [0], which are not turned on by default in the Murano Tempest
gate due to resource constraints.

However, I'm hesitant to switch to Heat's testing client, because it is not a
Tempest client, but rather the python-heatclient. I would like to know
whether there are plans to change this to a Tempest-based client?

There are no plans to switch the heat integration/functional tests to using
the tempest based client. The heat tests will use heatclient for most tests,
and gabbi for testing the REST API.

Since you're testing Murano rather than the Heat API, I think converting your
tests to heatclient would be reasonable.

I think that a Tempest-based Heat client should live somewhere anyway, for the same reasons which lead to the creation of the Tempest clients.

Even if as a Heat team you are not directly interested in it, other consumers may be interested (like in this case). I think that a Heat Tempest client should live somewhere anyway.

--
Luigi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 2, 2017 by Luigi_Toscano (1,700 points)   1 2
0 votes

On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects
which are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova,
glance, swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for Heat
/ orchestration related
configuration items in Tempest [4], and I would like to make sure that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end
the Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I
don't know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service
client manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and
service client out of Tempest.

We probably are mixing two different things here.

  1. Moving in-tree heat templest plugn and tests to a dedicated repo

Though we don't have any plans for it now, we may have to do it when/if
it's accepted as a community goal.

  1. Moving tempest tree heat tests and heat service client to a new home
    and owner.

I don't think that's something heat team would like to do given that we
don't use these tests anywhere and would probably spend time improving the
coverage of the gabbi api tests we already have.

Ok, well if the heat team has no interest in maintaining these tests there is
no point in keeping them around anymore. I've pushed up:

https://review.openstack.org/461841

to remove the tests. As for the clients we can just move those to tempest.lib
to not break the plugins that are using them.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 2, 2017 by Matthew_Treinish (11,200 points)   2 5 6
0 votes

On Tue, May 2, 2017 at 5:33 PM Matthew Treinish mtreinish@kortar.org
wrote:

On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:

On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
andrea.frittoli@gmail.com>
wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com
wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects
which are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova,
glance, swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure
(or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service
client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for
Heat
/ orchestration related
configuration items in Tempest [4], and I would like to make sure
that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end
the Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I
don't know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running
the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge
the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup,
as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want
to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service
client manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing
tempest
tests and service clients to a separate repo managed by heat team.
Though
that would be collective decision, I'm not sure that's something I
would
like to do. To start with we may look at adding some of the missing
pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest
to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion
about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and
service client out of Tempest.

We probably are mixing two different things here.

  1. Moving in-tree heat templest plugn and tests to a dedicated repo

Though we don't have any plans for it now, we may have to do it when/if
it's accepted as a community goal.

  1. Moving tempest tree heat tests and heat service client to a new home
    and owner.

I don't think that's something heat team would like to do given that we
don't use these tests anywhere and would probably spend time improving
the
coverage of the gabbi api tests we already have.

The test coverage provided by those tests is not available anywhere else at
the moment, however the tests are not really used, so I'm confused about
this.

You could run the existing Tempest tests with very little extra effort
until the
corresponding coverage is available in Gabbi format.

Ok, well if the heat team has no interest in maintaining these tests there
is
no point in keeping them around anymore. I've pushed up:

https://review.openstack.org/461841

Thanks for this.
We will need Heat PTL / QA Liaison +1 on the test removal before it goes
through.

to remove the tests. As for the clients we can just move those to
tempest.lib
to not break the plugins that are using them.

We could, but other projects maintain their own service clients, I'm not
convinced we should
make a special case for Heat.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 3, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote:
On Tue, May 2, 2017 at 5:33 PM Matthew Treinish mtreinish@kortar.org
wrote:

On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:

On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
andrea.frittoli@gmail.com>
wrote:

On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra ramishra@redhat.com
wrote:

On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
andrea.frittoli@gmail.com> wrote:

Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects
which are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova,
glance, swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure
(or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service
client
for the Heat project.
The Heat service client is used by the tests in Tempest, which run in
Heat gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the
layer4 job.
According to code search [3] the Heat service client is also used by
Murano and Daisycore.

For the heat grenade job, I've proposed two patches.

  1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
    phase

https://review.openstack.org/#/c/460542/

  1. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/

I proposed a patch to Tempest to start the deprecation counter for
Heat
/ orchestration related
configuration items in Tempest [4], and I would like to make sure
that
all tests and the service client
either find a new home outside of Tempest, or are removed, by the end
the Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I
don't know if those provide
enough coverage to replace the tests on Tempest side.

Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources), but I think that should not stop us from not running
the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge
the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup,
as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /

maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want
to
use the Heat service
client in their tests, even if the client is removed from Tempest, it
would still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service
client manager and be available
for use as today.

if I understand correctly, you're proposing moving the existing
tempest
tests and service clients to a separate repo managed by heat team.
Though
that would be collective decision, I'm not sure that's something I
would
like to do. To start with we may look at adding some of the missing
pieces
in heat tree itself.

I'm proposing to move tests and the service client outside of tempest
to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion
about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo is not a precondition for moving tests and
service client out of Tempest.

We probably are mixing two different things here.

  1. Moving in-tree heat templest plugn and tests to a dedicated repo

Though we don't have any plans for it now, we may have to do it when/if
it's accepted as a community goal.

  1. Moving tempest tree heat tests and heat service client to a new home
    and owner.

I don't think that's something heat team would like to do given that we
don't use these tests anywhere and would probably spend time improving
the
coverage of the gabbi api tests we already have.

The test coverage provided by those tests is not available anywhere else at
the moment, however the tests are not really used, so I'm confused about
this.

You could run the existing Tempest tests with very little extra effort
until the
corresponding coverage is available in Gabbi format.

Ok, well if the heat team has no interest in maintaining these tests there
is
no point in keeping them around anymore. I've pushed up:

https://review.openstack.org/461841

Thanks for this.
We will need Heat PTL / QA Liaison +1 on the test removal before it goes
through.

to remove the tests. As for the clients we can just move those to
tempest.lib
to not break the plugins that are using them.

We could, but other projects maintain their own service clients, I'm not
convinced we should
make a special case for Heat.

I'm not viewing it as a special case for heat, more treating it like any other
tempest interface that has a number of external consumers already but isn't in
lib yet. I don't think it's at all unreasonable to maintain the 321 lines of
heat client code in tempest.lib so that the plugins identified on this this
thread (and any others out there) don't lose an interface they rely on. The
maintenance burden for the clients is not high at all, especially since the
Rest API is supposed to be stable interface nothing should really need to
change in the client code often. We won't be adding new features just
preserving the existing interfaces. We just will need to document that they're
not self tested anymore because the tests no longer exist.

Also, like it or not it this already is a special case. This is the first and
only time a project team didn't take ownership of their project's tempest tests
since we made the decision to migrate higher level services into plugins back
at the Liberty summit. (it's also the last project left as part of that
migration)

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 4, 2017 by Matthew_Treinish (11,200 points)   2 5 6
...