settingsLogin | Registersettings

[openstack-dev] [cinder] Proposal: changes to our current testing process

0 votes

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests must
    be quite simple and very fast. Unit tests which takes more than 3-5 seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in a
    future to not ignore it. For now, there is not simple way to check coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 2, 2016 in openstack-dev by Ivan_Kolodyazhny (3,680 points)   5 8

23 Responses

0 votes

Hi Ivan!

On Wed, Mar 2, 2016 at 1:25 PM, Ivan Kolodyazhny e0ne@e0ne.info wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests
    must be quite simple and very fast. Unit tests which takes more than 3-5
    seconds should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in a
    future to not ignore it. For now, there is not simple way to check coverage
    because 'tox -e cover' output is not useful [2].

There is a script in Rally which can help you to check coverage rate -
https://github.com/openstack/rally/blob/master/tests/ci/cover.sh

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Andrey_Kurilin (3,820 points)   2 4
0 votes

+1 to making the testing process better.
It has been discussed that services could/should consider devoting some or all of a release cycle to stability and/or quality.
I propose the Cinder team makes improving and fixing the tests and test process a priority for the Newton cycle.

Scott D'Angelo (scottda)


From: Ivan Kolodyazhny [e0ne@e0ne.info]
Sent: Wednesday, March 02, 2016 4:25 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [cinder] Proposal: changes to our current testing process

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process better. I won't cover "3rd party CI's" topic here. I will share my opinion about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests must be quite simple and very fast. Unit tests which takes more than 3-5 seconds should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates. Usually, we require to add some unit-tests during code review process. Why can't we add coverage job to our CI and do not merge new patches, with will decrease tests coverage rate? Maybe, such job could be voting in a future to not ignore it. For now, there is not simple way to check coverage because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to infra to add new job [4]. Because these tests were moved from unit-tests, I think we're OK to make this job voting. Such tests should not be a replacement for Tempest. They even could tests Cinder with Fake Driver to make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can run them even with Cinder Fake Driver to make them not depended on a storage backend and make it faster. I believe, we can make this job voting soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other python-cinderclient consumers with a next merged patch. There is a thread in openstack-dev ML about such tests [8] and proposal [9] to introduce them to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like 'improves performance', 'fixes concurrency issues', etc. Even if we as a Cinder community don't have enough time to implement them, we have to ask for them in reviews, openstack-dev ML, file Rally bugs and blueprints if needed.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by scott.dangelo_at_hpe (960 points)   1 1
0 votes

That would be great. The boot from volume test remains one of the top
failure scenarios, both in the gate, and for end users. It would be
really good to get a handle on why and get that shored up.

-Sean

On 03/02/2016 08:18 AM, D'Angelo, Scott wrote:
+1 to making the testing process better.
It has been discussed that services could/should consider devoting some or all of a release cycle to stability and/or quality.
I propose the Cinder team makes improving and fixing the tests and test process a priority for the Newton cycle.

Scott D'Angelo (scottda)


From: Ivan Kolodyazhny [e0ne@e0ne.info]
Sent: Wednesday, March 02, 2016 4:25 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [cinder] Proposal: changes to our current testing process

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process better. I won't cover "3rd party CI's" topic here. I will share my opinion about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests must be quite simple and very fast. Unit tests which takes more than 3-5 seconds should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates. Usually, we require to add some unit-tests during code review process. Why can't we add coverage job to our CI and do not merge new patches, with will decrease tests coverage rate? Maybe, such job could be voting in a future to not ignore it. For now, there is not simple way to check coverage because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to infra to add new job [4]. Because these tests were moved from unit-tests, I think we're OK to make this job voting. Such tests should not be a replacement for Tempest. They even could tests Cinder with Fake Driver to make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can run them even with Cinder Fake Driver to make them not depended on a storage backend and make it faster. I believe, we can make this job voting soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other python-cinderclient consumers with a next merged patch. There is a thread in openstack-dev ML about such tests [8] and proposal [9] to introduce them to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like 'improves performance', 'fixes concurrency issues', etc. Even if we as a Cinder community don't have enough time to implement them, we have to ask for them in reviews, openstack-dev ML, file Rally bugs and blueprints if needed.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Sean_Dague (66,200 points)   4 12 22
0 votes

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests must
    be quite simple and very fast. Unit tests which takes more than 3-5 seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in a
    future to not ignore it. For now, there is not simple way to check coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Eric_Harney (2,100 points)   1 2
0 votes

Eric,

There are Gorka's patches [10] to remove API Races

[10]
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney eharney@redhat.com wrote:

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my
opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests
    must
    be quite simple and very fast. Unit tests which takes more than 3-5
    seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review
    process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in
    a
    future to not ignore it. For now, there is not simple way to check
    coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from
unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we
can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job
voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce
them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Ivan_Kolodyazhny (3,680 points)   5 8
0 votes

On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
Eric,

There are Gorka's patches [10] to remove API Races

[10]
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

So the second part of my question is, is writing a Rally job to prove
out that code a reasonable task?

How hard is that to do and what does it look like?

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney eharney@redhat.com wrote:

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my
opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests
    must
    be quite simple and very fast. Unit tests which takes more than 3-5
    seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review
    process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in
    a
    future to not ignore it. For now, there is not simple way to check
    coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from
unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we
can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job
voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce
them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Eric_Harney (2,100 points)   1 2
0 votes

Rally is not part of the gate.
Also making performance test without 3rd party CI will not be very useful.
It is a good idea to run Rally performance and scenario testing but outside gate process.

From: Ivan Kolodyazhny [mailto:e0ne@e0ne.info]
Sent: Wednesday, March 02, 2016 8:36 AM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

Eric,

There are Gorka's patches [10] to remove API Races

[10] https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney eharney@redhat.com wrote:
On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests must
    be quite simple and very fast. Unit tests which takes more than 3-5 seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in a
    future to not ignore it. For now, there is not simple way to check coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Arkady_Kanevsky_at_D (1,480 points)   2
0 votes

Eric,

For now, we test Cinder API with some concurrency only with Rally, so, IMO,
it's reasonable get more scenarios for API races fixes.

It's not a hard task to implement new scenarios, they are pretty simple:
[11] and [12]

[11]
https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
[12]
https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney eharney@redhat.com wrote:

On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:

Eric,

There are Gorka's patches [10] to remove API Races

[10]

https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

So the second part of my question is, is writing a Rally job to prove
out that code a reasonable task?

How hard is that to do and what does it look like?

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney eharney@redhat.com wrote:

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my
opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests
    must
    be quite simple and very fast. Unit tests which takes more than 3-5
    seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review
    process. Why
    can't we add coverage job to our CI and do not merge new patches,
    with
    will decrease tests coverage rate? Maybe, such job could be voting
    in
    a
    future to not ignore it. For now, there is not simple way to check
    coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from
unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver
to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we
can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job
voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a
thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce
them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches
like
'improves performance', 'fixes concurrency issues', etc. Even if we as
a
Cinder community don't have enough time to implement them, we have to
ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints
if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the
patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Ivan_Kolodyazhny (3,680 points)   5 8
0 votes

Arkady,

It's not true. We've got non-voting Rally job on Cinder gates called
"gate-rally-dsvm-cinder".

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:52 PM, Arkady_Kanevsky@dell.com wrote:

Rally is not part of the gate.

Also making performance test without 3rd party CI will not be very useful.

It is a good idea to run Rally performance and scenario testing but
outside gate process.

From: Ivan Kolodyazhny [mailto:e0ne@e0ne.info]
Sent: Wednesday, March 02, 2016 8:36 AM
To: OpenStack Development Mailing List (not for usage questions) <
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [cinder] Proposal: changes to our current
testing process

Eric,

There are Gorka's patches [10] to remove API Races

[10]
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney eharney@redhat.com wrote:

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my
opinion
about current and feature jobs.

Unit-tests

  • Long-running tests. I hope, everybody will agree that unit-tests
    must
    be quite simple and very fast. Unit tests which takes more than 3-5
    seconds
    should be refactored and/or moved to 'integration' tests.
    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
    good to have some hacking checks to prevent such issues in a future.

  • Tests coverage. We don't check it in an automatic way on gates.

    Usually, we require to add some unit-tests during code review
    process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in
    a
    future to not ignore it. For now, there is not simple way to check
    coverage
    because 'tox -e cover' output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from
unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we
can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job
voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce
them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Ivan_Kolodyazhny (3,680 points)   5 8
0 votes

On 02/03, Ivan Kolodyazhny wrote:
Eric,

There are Gorka's patches [10] to remove API Races

[10]
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

I looked at Rally a long time ago so apologies if I'm totally off base
here, but it looked like it was a performance evaluation tool, which
means that it probably won't help to check for API Races (at least I
didn't see how when I looked).

Many of the API races only happen if you simultaneously try the same
operation multiple times against the same resource or if there are
different operations that are trying to operate on the same resource.

On the first case if Rally allowed it we could test it because we know
only 1 of the operations should succeed, but on the second case when we
are talking about preventing races from different operations there is no
way to know what the result should be, since the order in which those
operations are executed on each test run will determine which one will
fail and which one will succeed.

I'm not trying to go against the general idea of adding rally tests, I
just think that they won't help in the case of the API races.

[ ... ]

  • Tests coverage. We don't check it in an automatic way on gates.
    Usually, we require to add some unit-tests during code review
    process. Why
    can't we add coverage job to our CI and do not merge new patches, with
    will decrease tests coverage rate? Maybe, such job could be voting in
    a
    future to not ignore it. For now, there is not simple way to check
    coverage
    because 'tox -e cover' output is not useful [2].

In my experience preventing patches that reduce test coverage can be
sometimes problematic because it's a percentage, and many times code
refactoring gives false positives. :-(

Cheers,
Gorka.

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from
unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.

Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.

Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].

Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we
can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job
voting
soon. Also, we need more contributors to this kind of tests.

Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce
them
to python-cinderclient.

Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 2, 2016 by Gorka_Eguileor (2,960 points)   2 4
...