settingsLogin | Registersettings

[openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

0 votes

Hi All,

As you might already know, within Red Hat's tempest fork, we do have one
tempest configuration script which was built in the past by David Kranz
[1] and that's been actively used in our CI system. Regarding this
topic, I'm aware that quite some effort has been done in the past [2]
and I would like to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not
the tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit in the QA Priorities meting [3]. I propose we use the existing
etherpad to continue the discussion around and tracking of implementing
"tempest config-create" using the downstream config script as a starting
point. [4]

If you have any questions, comments or opinion, please let me know.

Best Regards

Daniel Mellado


[1]
https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
[2] https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
[3] https://etherpad.openstack.org/p/mitaka-qa-priorities
[4] https://etherpad.openstack.org/p/tempest-cli-improvements

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 24, 2015 in openstack-dev by Daniel_Mellado (820 points)  

19 Responses

0 votes

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have one
tempest configuration script which was built in the past by David Kranz [1]
and that's been actively used in our CI system. Regarding this topic, I'm
aware that quite some effort has been done in the past [2] and I would like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka summit
in the QA Priorities meting [3]. I propose we use the existing etherpad to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting point. [4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi

responded Nov 26, 2015 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator in
Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have one
tempest configuration script which was built in the past by David Kranz
[1]
and that's been actively used in our CI system. Regarding this topic, I'm
aware that quite some effort has been done in the past [2] and I would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing etherpad
to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 26, 2015 by Andrey_Kurilin (3,820 points)   2 4
0 votes

Hello everyone,

Yes, I am working on this now. We have some success already, but there is a
lot of work to do. Of course, some things don't work ideally. For example,
in [2] from the previous letter we have not 24 skipped tests, actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator
in Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have one
tempest configuration script which was built in the past by David Kranz
[1]
and that's been actively used in our CI system. Regarding this topic,
I'm
aware that quite some effort has been done in the past [2] and I would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing etherpad
to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 26, 2015 by Yaroslav_Lobankov (740 points)   2
0 votes

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
- failed: 0

[1] -
http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but there is
a lot of work to do. Of course, some things don't work ideally. For
example, in [2] from the previous letter we have not 24 skipped tests,
actually much more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator
in Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this topic,
I'm
aware that quite some effort has been done in the past [2] and I would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 26, 2015 by Andrey_Kurilin (3,820 points)   2 4
0 votes

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they should
at least know their environment, this script would set a minimum
acceptable default. Do you think PTL and Pre-PTL concerns that we spoke
of would still apply to that scenario?

Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
should enable them)
- failed: 0

[1] -
http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov
<ylobankov@mirantis.com ylobankov@mirantis.com> wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but
there is a lot of work to do. Of course, some things don't work
ideally. For example, in [2] from the previous letter we have not
24 skipped tests, actually much more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.  

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin
<akurilin@mirantis.com <mailto:akurilin@mirantis.com>> wrote:

    Hi!
    Boris P. and I tried to push a spec[1] for automation tempest
    config generator, but we did not succeed to merge it. Imo,
    qa-team doesn't want to have such tool:(

    >However, there is a big concern:
    >If the script contain a bug and creates the configuration
    which makes
    >most tests skipped, we cannot do enough tests on the gate.
    >Tempest contains 1432 tests and difficult to detect which
    tests are
    >skipped as unexpected.

    Yaroslav Lobankov is working on improvement for tempest config
    generator in Rally. Last time when we launch full tempest
    run[2], we got 1154 success tests and only 24 skipped. Also,
    there is a patch, which adds x-fail mechanism(it based on
    subunit-filter): you can transmit a file with test names +
    reasons and rally will modify results.

    [1] - https://review.openstack.org/#/c/94473/

    [2] -
    http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

    On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi
    <ken1ohmichi@gmail.com <mailto:ken1ohmichi@gmail.com>> wrote:

        Hi Daniel,

        Thanks for pointing this up.

        2015-11-25 1:40 GMT+09:00 Daniel Mellado
        <daniel.mellado.es@ieee.org
        <mailto:daniel.mellado.es@ieee.org>>:
        > Hi All,
        >
        > As you might already know, within Red Hat's tempest
        fork, we do have one
        > tempest configuration script which was built in the past
        by David Kranz [1]
        > and that's been actively used in our CI system.
        Regarding this topic, I'm
        > aware that quite some effort has been done in the past
        [2] and I would like
        > to complete the implementation of this blueprint/spec.
        >
        > My plan would be to have this script under the
        /tempest/cmd or
        > /tempest/tools folder from tempest so it can be used to
        configure not the
        > tempest gate but any cloud we'd like to run tempest against.
        >
        > Adding the configuration script was discussed briefly at
        the Mitaka summit
        > in the QA Priorities meting [3]. I propose we use the
        existing etherpad to
        > continue the discussion around and tracking of
        implementing "tempest
        > config-create" using the downstream config script as a
        starting point. [4]
        >
        > If you have any questions, comments or opinion, please
        let me know.

        This topic have happened several times, and I also felt
        this kind of
        tool was very useful for Tempest users, because Tempest
        contains 296
        options($ grep cfg * -R | grep Opt | wc -l) now and it is
        difficult to
        set the configuration up.
        However, there is a big concern:
        If the script contain a bug and creates the configuration
        which makes
        most tests skipped, we cannot do enough tests on the gate.
        Tempest contains 1432 tests and difficult to detect which
        tests are
        skipped as unexpected.
        Actually we faced unexpected skipped tests on the gate
        before due to
        some bug, then the problem has been fixed.
        But I can imagine this kind of problem happens after
        implementing this
        kind of script.

        So now I am feeling Tempest users need to know what cloud
        they want to
        test with Tempest, and need to know what tests run with
        Tempest.
        Testers need to know what test target/items they are
        testing basically.

        Thanks
        Ken Ohmichi

        ---

        > ---
        > [1]
        >
        https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
        > [2]
        https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
        > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
        > [4]
        https://etherpad.openstack.org/p/tempest-cli-improvements
        >
        >
        https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
        >
        >
        __________________________________________________________________________
        > OpenStack Development Mailing List (not for usage questions)
        > Unsubscribe:
        OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
        
        >
        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
        >

        __________________________________________________________________________
        OpenStack Development Mailing List (not for usage questions)
        Unsubscribe:
        OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
        
        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




    -- 
    Best regards,
    Andrey Kurilin.

    __________________________________________________________________________
    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
    
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 27, 2015 by Daniel_Mellado (820 points)  
0 votes

2015-11-27 15:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's still
good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they should at
least know their environment, this script would set a minimum acceptable
default. Do you think PTL and Pre-PTL concerns that we spoke of would still
apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi


Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
- failed: 0

[1] -
http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov ylobankov@mirantis.com
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but there is
a lot of work to do. Of course, some things don't work ideally. For example,
in [2] from the previous letter we have not 24 skipped tests, actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator
in Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this topic,
I'm
aware that quite some effort has been done in the past [2] and I would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 27, 2015 by Ken'ichi_Ohmichi (8,640 points)   2 3 3
0 votes

Hi,
I think this script is valuable to some users: Rally and Red Hat expressed
their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs,
Tempest could be misconfigured. So, we could have this tool inside the
Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use
this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

2015-11-27 15:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still
good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they should
at
least know their environment, this script would set a minimum acceptable
default. Do you think PTL and Pre-PTL concerns that we spoke of would
still
apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi


Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
- failed: 0

[1] -

http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz

[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
ylobankov@mirantis.com>
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but there
is
a lot of work to do. Of course, some things don't work ideally. For
example,
in [2] from the previous letter we have not 24 skipped tests, actually
much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't
want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config
generator
in Rally. Last time when we launch full tempest run[2], we got 1154
success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with
test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -

http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
ken1ohmichi@gmail.com>
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es@ieee.org
:

Hi All,

As you might already know, within Red Hat's tempest fork, we do have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this
topic,
I'm
aware that quite some effort has been done in the past [2] and I
would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure
not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing "tempest
config-create" using the downstream config script as a starting
point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing
basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

[2]

https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 27, 2015 by Jordan_Pittier (3,060 points)   2 3
0 votes

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <jordan.pittier@scality.com
wrote:

Hi,
I think this script is valuable to some users: Rally and Red Hat expressed
their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs,
Tempest could be misconfigured. So, we could have this tool inside the
Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use
this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

2015-11-27 15:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still
good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they
should at
least know their environment, this script would set a minimum acceptable
default. Do you think PTL and Pre-PTL concerns that we spoke of would
still
apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi


Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
- failed: 0

[1] -

http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz

[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
ylobankov@mirantis.com>
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but there
is
a lot of work to do. Of course, some things don't work ideally. For
example,
in [2] from the previous letter we have not 24 skipped tests, actually
much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin@mirantis.com

wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't
want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config
generator
in Rally. Last time when we launch full tempest run[2], we got 1154
success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with
test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -

http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
ken1ohmichi@gmail.com>
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es@ieee.org
:

Hi All,

As you might already know, within Red Hat's tempest fork, we do
have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this
topic,
I'm
aware that quite some effort has been done in the past [2] and I
would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure
not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing
"tempest
config-create" using the downstream config script as a starting
point.
[4]

If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult
to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing
this
kind of script.

So now I am feeling Tempest users need to know what cloud they want
to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing
basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

[2]

https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 29, 2015 by Yair_Fried (800 points)   1
0 votes

Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know "what
do we test for?" and we shouldn't discover values that we test for
automatically.
If users seems that "My current cloud is good. This is what I expect.",
discovering function could work. But I suppose many of users would use
tempest-config-generator for a new their cloud. So I feel the tool users
could be misunderstanding easily.

But I also think that tempest users don't need to know all of the
configurations.
So, how about something like introducing "a configuration wizard" for
tempest configuration?
This is a just idea, though..

Anyway, if you have the motivation to introduce tempest-config, how about
writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users,
solutions, etc.)

Best Regards,
-- Masayuki Igawa

2015年11月29日(日) 22:07 Yair Fried yfried@redhat.com:

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
jordan.pittier@scality.com> wrote:

Hi,
I think this script is valuable to some users: Rally and Red Hat
expressed their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of
bugs, Tempest could be misconfigured. So, we could have this tool inside
the Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use
this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi ken1ohmichi@gmail.com
wrote:

2015-11-27 15:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still
good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest
options?
As you pointed out there are quite a few of them and even if they
should at
least know their environment, this script would set a minimum
acceptable
default. Do you think PTL and Pre-PTL concerns that we spoke of would
still
apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi


Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it
to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
should
enable them)
- failed: 0

[1] -

http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz

[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
ylobankov@mirantis.com>
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but
there is
a lot of work to do. Of course, some things don't work ideally. For
example,
in [2] from the previous letter we have not 24 skipped tests,
actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <
akurilin@mirantis.com>
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't
want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which
makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config
generator
in Rally. Last time when we launch full tempest run[2], we got 1154
success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with
test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -

http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
ken1ohmichi@gmail.com>
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado <
daniel.mellado.es@ieee.org>:

Hi All,

As you might already know, within Red Hat's tempest fork, we do
have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this
topic,
I'm
aware that quite some effort has been done in the past [2] and I
would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure
not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the
Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing
"tempest
config-create" using the downstream config script as a starting
point.
[4]

If you have any questions, comments or opinion, please let me
know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult
to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which
makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing
this
kind of script.

So now I am feeling Tempest users need to know what cloud they want
to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing
basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

[2]

https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
-- Masayuki Igawa


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 30, 2015 by Masayuki_Igawa (1,240 points)   1
0 votes

Looking at the tool, it seems to me that is servers a combination of
functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test
resources, and have a CLI tool to provision them [0]. We could re-use code
from this tool to achieve that.

Support for distribution specific / cloud specific overrides is also
something that is useful. In clouds where I control the deployment process
I inject extra configs in tempest.conf based on the deployment options. In
clouds where I don't, I maintain a partial tempest.conf with the list of
options which I expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to
the bottom of tempest.conf and it's done. Duplicate configuration options
are not an issue, the last one wins. Still we could support specifying a
number of configuration files to the non-yet implemented "tempest run"
command.

Support for configuration override via CLI is something that I can see it
can be useful during development or troubleshooting, we could support that
as options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use
it today in the gate - i.e. fail fast if the generated configuration does
not match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the
functions implemented by it can fit into tempest - and some are already
there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa masayuki.igawa@gmail.com
wrote:

Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know
"what do we test for?" and we shouldn't discover values that we test for
automatically.
If users seems that "My current cloud is good. This is what I expect.",
discovering function could work. But I suppose many of users would use
tempest-config-generator for a new their cloud. So I feel the tool users
could be misunderstanding easily.

But I also think that tempest users don't need to know all of the
configurations.
So, how about something like introducing "a configuration wizard" for
tempest configuration?
This is a just idea, though..

Anyway, if you have the motivation to introduce tempest-config, how about
writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users,
solutions, etc.)

Best Regards,
-- Masayuki Igawa

2015年11月29日(日) 22:07 Yair Fried yfried@redhat.com:

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience
is people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
jordan.pittier@scality.com> wrote:

Hi,
I think this script is valuable to some users: Rally and Red Hat
expressed their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of
bugs, Tempest could be misconfigured. So, we could have this tool inside
the Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t
use this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi <ken1ohmichi@gmail.com

wrote:

2015-11-27 15:40 GMT+09:00 Daniel Mellado daniel.mellado.es@ieee.org:

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still
good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested
on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest
options?
As you pointed out there are quite a few of them and even if they
should at
least know their environment, this script would set a minimum
acceptable
default. Do you think PTL and Pre-PTL concerns that we spoke of would
still
apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi


Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it
to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:

Sorry for wrong numbers. The bug-fix for issue with counters is
merged.
Correct numbers(latest result from rally's gate[1]):
- total number of executed tests: 1689
- success: 1155
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
should
enable them)
- failed: 0

[1] -

http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz

[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
ylobankov@mirantis.com>
wrote:

Hello everyone,

Yes, I am working on this now. We have some success already, but
there is
a lot of work to do. Of course, some things don't work ideally. For
example,
in [2] from the previous letter we have not 24 skipped tests,
actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <
akurilin@mirantis.com>
wrote:

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't
want to
have such tool:(

However, there is a big concern:
If the script contain a bug and creates the configuration which
makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config
generator
in Rally. Last time when we launch full tempest run[2], we got 1154
success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with
test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -

http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
ken1ohmichi@gmail.com>
wrote:

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado <
daniel.mellado.es@ieee.org>:

Hi All,

As you might already know, within Red Hat's tempest fork, we do
have
one
tempest configuration script which was built in the past by David
Kranz [1]
and that's been actively used in our CI system. Regarding this
topic,
I'm
aware that quite some effort has been done in the past [2] and I
would
like
to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to
configure not
the
tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the
Mitaka
summit
in the QA Priorities meting [3]. I propose we use the existing
etherpad to
continue the discussion around and tracking of implementing
"tempest
config-create" using the downstream config script as a starting
point.
[4]

If you have any questions, comments or opinion, please let me
know.

This topic have happened several times, and I also felt this kind
of
tool was very useful for Tempest users, because Tempest contains
296
options($ grep cfg * -R | grep Opt | wc -l) now and it is
difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which
makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due
to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing
this
kind of script.

So now I am feeling Tempest users need to know what cloud they
want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing
basically.

Thanks
Ken Ohmichi



[1]

https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py

[2]

https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
>
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
-- Masayuki Igawa


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 30, 2015 by Andrea_Frittoli (5,920 points)   1 2 3
...