settingsLogin | Registersettings

[openstack-dev] [QA] The end-user test suite for OpenStack clusters

0 votes

Hi team,

we have an idea to create the test suite with destructive/HA and advanced
end-user scenarios for the OpenStack clusters. This test suite will
contains advanced scenario integration tests for OpenStack clusters to make
sure that the cluster is ready for the production.

The test cases which we want to cover in this test suite:
1) All simple and advanced destructive actions, like a reboot of the nodes,
restart OpenStack services and etc. (we can probably use of-faults library
[1], which we already use in Rally)
2) All advanced test scenarios like a migration of the bunch of VMs between
nodes and booting of the VMs with large images (10+ Gb), send traffic
between VMs and in parallel restart Neutron l3 agents and etc.

The key requirements:
1) The framework should know the details of the deployments (how many nodes
we have, how to ssh to OpenStack nodes, how to restart the nodes and etc.).
This is why we don't want to add such "advanced" and HA-focused test
scenarios to Tempest.
2) We should be ready to run these tests for any clouds: DevStack clouds
(we can skip HA cases for DevStack), Fuel clouds, clouds which were
deployed by Ansible or Puppet tools.
3) This framework should allow reproduce the issue in a repeatable manner,
this is why we can't just cover all the tests with Rally load tests +
destructive plugins (we are working on this right now too to have an
ability to test HA-related scenarios under the load).

As we discussed on the OpenStack summit a year ago it is better to move
such test suite in a separate repository and this framework can became a
part of the QA (or at least BigTent) program in OpenStack.

I've created the commit to OpenStack project-config repository:
https://review.openstack.org/#/c/374667/

Could you please take a look?

We understand that it will be hard to add such test suite to the gates for
every commit in OpenStack because we will need a lot of hardware. We don't
want to add these tests to the per-commit gates for now, it is ok to run
them just once a day, for example. And we definitely need to have such test
suite to validate our own pre-production clouds.

Thank you!

[1] https://github.com/openstack/os-faults

--

Timur,
QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Feb 28, 2017 in openstack-dev by Timur_Nurlygayanov (2,440 points)   1 4 8
retagged Jan 26, 2017 by admin

14 Responses

0 votes

Hi Timur,

Thanks for picking this up, that is interesting for me.

2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

we have an idea to create the test suite with destructive/HA and advanced
end-user scenarios for the OpenStack clusters. This test suite will contains
advanced scenario integration tests for OpenStack clusters to make sure that
the cluster is ready for the production.

The test cases which we want to cover in this test suite:
1) All simple and advanced destructive actions, like a reboot of the nodes,
restart OpenStack services and etc. (we can probably use of-faults library
1, which we already use in Rally)
2) All advanced test scenarios like a migration of the bunch of VMs between
nodes and booting of the VMs with large images (10+ Gb), send traffic
between VMs and in parallel restart Neutron l3 agents and etc.

The key requirements:
1) The framework should know the details of the deployments (how many nodes
we have, how to ssh to OpenStack nodes, how to restart the nodes and etc.).
This is why we don't want to add such "advanced" and HA-focused test
scenarios to Tempest.

Yeah, this point is right. This "advanced" way is different from the
design principle of Tempest1.
I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?
For productions(or distributions), this verifying point seems
important because service scripts need to restart OpenStack services
automatically.
But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Thanks
Ken Ohmichi


2) We should be ready to run these tests for any clouds: DevStack clouds (we
can skip HA cases for DevStack), Fuel clouds, clouds which were deployed by
Ansible or Puppet tools.
3) This framework should allow reproduce the issue in a repeatable manner,
this is why we can't just cover all the tests with Rally load tests +
destructive plugins (we are working on this right now too to have an ability
to test HA-related scenarios under the load).

As we discussed on the OpenStack summit a year ago it is better to move such
test suite in a separate repository and this framework can became a part of
the QA (or at least BigTent) program in OpenStack.

I've created the commit to OpenStack project-config repository:
https://review.openstack.org/#/c/374667/

Could you please take a look?

We understand that it will be hard to add such test suite to the gates for
every commit in OpenStack because we will need a lot of hardware. We don't
want to add these tests to the per-commit gates for now, it is ok to run
them just once a day, for example. And we definitely need to have such test
suite to validate our own pre-production clouds.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 27, 2016 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Hi Ken,

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

Thank you!

On Wed, Sep 28, 2016 at 2:37 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for picking this up, that is interesting for me.

2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

we have an idea to create the test suite with destructive/HA and advanced
end-user scenarios for the OpenStack clusters. This test suite will
contains
advanced scenario integration tests for OpenStack clusters to make sure
that
the cluster is ready for the production.

The test cases which we want to cover in this test suite:
1) All simple and advanced destructive actions, like a reboot of the
nodes,
restart OpenStack services and etc. (we can probably use of-faults
library
[1], which we already use in Rally)
2) All advanced test scenarios like a migration of the bunch of VMs
between
nodes and booting of the VMs with large images (10+ Gb), send traffic
between VMs and in parallel restart Neutron l3 agents and etc.

The key requirements:
1) The framework should know the details of the deployments (how many
nodes
we have, how to ssh to OpenStack nodes, how to restart the nodes and
etc.).
This is why we don't want to add such "advanced" and HA-focused test
scenarios to Tempest.

Yeah, this point is right. This "advanced" way is different from the
design principle of Tempest[1].
I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?
For productions(or distributions), this verifying point seems
important because service scripts need to restart OpenStack services
automatically.
But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Thanks
Ken Ohmichi


[1]: https://github.com/openstack/tempest#design-principles

2) We should be ready to run these tests for any clouds: DevStack clouds
(we
can skip HA cases for DevStack), Fuel clouds, clouds which were deployed
by
Ansible or Puppet tools.
3) This framework should allow reproduce the issue in a repeatable
manner,
this is why we can't just cover all the tests with Rally load tests +
destructive plugins (we are working on this right now too to have an
ability
to test HA-related scenarios under the load).

As we discussed on the OpenStack summit a year ago it is better to move
such
test suite in a separate repository and this framework can became a part
of
the QA (or at least BigTent) program in OpenStack.

I've created the commit to OpenStack project-config repository:
https://review.openstack.org/#/c/374667/

Could you please take a look?

We understand that it will be hard to add such test suite to the gates
for
every commit in OpenStack because we will need a lot of hardware. We
don't
want to add these tests to the per-commit gates for now, it is ok to run
them just once a day, for example. And we definitely need to have such
test
suite to validate our own pre-production clouds.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2016 by Timur_Nurlygayanov (2,440 points)   1 4 8
0 votes

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2016 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2016 by Yaroslav_Lobankov (740 points)   3
0 votes

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov <tnurlygayanov@mirantis.com

:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 6, 2016 by Timur_Nurlygayanov (2,440 points)   1 4 8
0 votes

Hi Timur,

2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

Sorry for my misleading, that is not I wanted to say.
I am thinking Tempest doesn't use os-faults as library.
I just wanted to say some destructive tests which will use REST APIs
can be implemented in Tempest instead of os-faults.

For example, the compute service client contains disable_service which
disables nova service but Tempest doesn't contain the corresponding
test cases because Tempest tests need to run in parallel.
https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54

If some tests use this method, the other tests which run in parallel
can be failed as unexpected on the gate.
Such tests also are destructive and I am hoping these tests also will
be able to run the gate by finding better way.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

OK, I'd like to summarize my thinking here for adding os-faults under
QA project:

Under QA project, the first target of most programs(tempest, devstack,
etc) is for the gate testing.
Of course, some of them are available on production clouds also as the
design, but the first is for the gate.
That means devstack clouds as the first target/purpose.
At this time, this os-faults doesn't seem the gate as the first
purpose based on current implementation.
So I feel os-faults seems different from the existing programs.

os-faults is very young and we will be able to extend it for devstack
clouds also later, I hope.
In addition, I heard this kind of tests(destructive, HA, etc) are
requested from the other users.
So this os-faults seems very valuable for users.

Just in my opinion, I am find to add this os-faults under QA project.
But before that, I'd like to get opinions from the other people of QA team.

Thanks
Ken Ohmichi

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2016 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Hi,

My first impression is "It's interesting" :)

I actually don't read whole of this threads, but, I re-read the QA
mission statement[1].
I feel os-faults is a little bit far from the mission statement.
Because os-faults seems like focusing on operator testing.

But I think this can be under the QA project. Because tempest scope
has also operator testing.

[1] https://wiki.openstack.org/wiki/QA#Project_Team_Definition

On Sat, Oct 8, 2016 at 4:43 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com wrote:
Hi Timur,

2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

Sorry for my misleading, that is not I wanted to say.
I am thinking Tempest doesn't use os-faults as library.
I just wanted to say some destructive tests which will use REST APIs
can be implemented in Tempest instead of os-faults.

For example, the compute service client contains disable_service which
disables nova service but Tempest doesn't contain the corresponding
test cases because Tempest tests need to run in parallel.
https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54

If some tests use this method, the other tests which run in parallel
can be failed as unexpected on the gate.
Such tests also are destructive and I am hoping these tests also will
be able to run the gate by finding better way.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

OK, I'd like to summarize my thinking here for adding os-faults under
QA project:

Under QA project, the first target of most programs(tempest, devstack,
etc) is for the gate testing.
Of course, some of them are available on production clouds also as the
design, but the first is for the gate.
That means devstack clouds as the first target/purpose.
At this time, this os-faults doesn't seem the gate as the first
purpose based on current implementation.
So I feel os-faults seems different from the existing programs.

os-faults is very young and we will be able to extend it for devstack
clouds also later, I hope.
In addition, I heard this kind of tests(destructive, HA, etc) are
requested from the other users.
So this os-faults seems very valuable for users.

Just in my opinion, I am find to add this os-faults under QA project.
But before that, I'd like to get opinions from the other people of QA team.

Thanks
Ken Ohmichi

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 13, 2016 by masayuki_at_igawa.me (700 points)  
0 votes

Hi Timur,

I discussed this on irc, and it seems hard to get feedback from many
people because now the scope and implementation of this os-faults are
separated into multiple pieces(like multiple mails, github code,
gerrit etc).
So how about proposing this as qa-spec?
If doing that, We can discuss this on clear document and it is clear
to get a consensus about adding this.

Thanks
Ken Ohmichi


.

2016-10-07 12:43 GMT-07:00 Ken'ichi Ohmichi ken1ohmichi@gmail.com:

Hi Timur,

2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

Sorry for my misleading, that is not I wanted to say.
I am thinking Tempest doesn't use os-faults as library.
I just wanted to say some destructive tests which will use REST APIs
can be implemented in Tempest instead of os-faults.

For example, the compute service client contains disable_service which
disables nova service but Tempest doesn't contain the corresponding
test cases because Tempest tests need to run in parallel.
https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54

If some tests use this method, the other tests which run in parallel
can be failed as unexpected on the gate.
Such tests also are destructive and I am hoping these tests also will
be able to run the gate by finding better way.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

OK, I'd like to summarize my thinking here for adding os-faults under
QA project:

Under QA project, the first target of most programs(tempest, devstack,
etc) is for the gate testing.
Of course, some of them are available on production clouds also as the
design, but the first is for the gate.
That means devstack clouds as the first target/purpose.
At this time, this os-faults doesn't seem the gate as the first
purpose based on current implementation.
So I feel os-faults seems different from the existing programs.

os-faults is very young and we will be able to extend it for devstack
clouds also later, I hope.
In addition, I heard this kind of tests(destructive, HA, etc) are
requested from the other users.
So this os-faults seems very valuable for users.

Just in my opinion, I am find to add this os-faults under QA project.
But before that, I'd like to get opinions from the other people of QA team.

Thanks
Ken Ohmichi

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 13, 2016 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

I like os-faults library which can provide the abstraction over different destructive actions like reboot/poweroff node etc.

But not much clear about Stepler that what all tests it will contain. Tempest do have scenario tests which can hit the production env with significant way of testing scenario.
It can cover the end user scenario also which involves the interaction of public OpenStack APIs and always in enhancement state by adding more and more scenario tests.

Few query over Stepler as separate project:

  1. Is Stepler will cover only tests which hits the node level action(reboot, HA etc)?

  2. This should not mix the scenario tests which are in Tempest scope because that can make confusion for developers (where to add scenario tests) as well as for tester(from where to run what scenario tests).

  3. How we make sure those tests run fine or at least run while adding.

a. I think devstack enhancement for multi-node etc can do this as mentioned by you also.

b. If so then why not adding those tests in Tempest only using os-faults lib ?

Overall I feel os-faults as lib is really nice idea but tests scope can go in Tempest under HW_scenario (or something else) till it does not break basic principle of Tempest.

Thanks
gmann

From: Timur Nurlygayanov [mailto:tnurlygayanov@mirantis.com]
Sent: 06 October 2016 20:09
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com wrote:
Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two months old), but you can find some examples of the use in the os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com wrote:
Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.
I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 14, 2016 by ghanshyam.mann_at_ne (940 points)   1
0 votes

Hi team,

here is a short update:

1) The QA user story for destructive testing of OpenStack cloud is on
review [1].
2) The spec for a new framework which will focus on HA/failover and
destructive testing is no review [2].
3) The commit for the new repository is on review [3] as well.

[1] https://review.openstack.org/#/c/396142
[2] https://review.openstack.org/#/c/399618
[3] https://review.openstack.org/#/c/374667

On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann <
ghanshyam.mann@nectechnologies.in> wrote:

I like os-faults library which can provide the abstraction over different
destructive actions like reboot/poweroff node etc.

But not much clear about Stepler that what all tests it will contain.
Tempest do have scenario tests which can hit the production env with
significant way of testing scenario.

It can cover the end user scenario also which involves the interaction of
public OpenStack APIs and always in enhancement state by adding more and
more scenario tests.

Few query over Stepler as separate project:

  1. Is Stepler will cover only tests which hits the node level
    action(reboot, HA etc)?

  2. This should not mix the scenario tests which are in Tempest
    scope because that can make confusion for developers (where to add scenario
    tests) as well as for tester(from where to run what scenario tests).

  3. How we make sure those tests run fine or at least run while
    adding.

a. I think devstack enhancement for multi-node etc can do this as
mentioned by you also.

b. If so then why not adding those tests in Tempest only using
os-faults lib ?

Overall I feel os-faults as lib is really nice idea but tests scope can
go in Tempest under HW_scenario (or something else) till it does not break
basic principle of Tempest.

Thanks

gmann

From: Timur Nurlygayanov [mailto:tnurlygayanov@mirantis.com]
Sent: 06 October 2016 20:09
To: OpenStack Development Mailing List (not for usage questions) <
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack
clusters

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able

to manage cluster nodes and OpenStack services on these nodes.

It is a good idea to add some Tempest tests which will use

os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive

actions in the clouds (the reboot of nodes, restart of OpenStack services,

enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with

advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,

Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov tnurlygayanov@mirantis.com:

I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification
of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are
targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,

Senior QA Manager

OpenStack Projects

Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
QA Manager
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 28, 2016 by Timur_Nurlygayanov (2,440 points)   1 4 8
...