settingsLogin | Registersettings

[Openstack-operators] [openstack-dev][all][qa][gabbi][rally][tempest] Extend "rally verfiy" to unify work with Gabbi, Tempest and all in-tree functional tests

0 votes

Hi stackers,

Intro (Gabbi)


Gabbi is amazing tool that allows you to describe in human readable way
what API requests to execute and what you are expecting as a result. It
Simplifies a lot API testing.

It's based on unittest so it can be easily run using tox/tester/nose and so
on..

Intro (Functional in-tree tests)


Keeping all tests in one project like Tempest, that is maintained by one
team, was not enough scalable approach. To scale things, projects started
maintaining their own functional tests in their own tree. This resolves
scale issues and now new features can be merged with functional tests.

The Problem


As far as you know there are a lot of OpenStack projects with their own
functional tests / gabbi tests in tree. It becomes hard for developers,
devops and operators to work with them. (Like it's hard to install
OpenStack by hands without DevStack. )

Usually, end users are choosing 2 approach:
1) Make own tests
2) Make scripts that runs somehow all these tests

Small Intro (Rally)


Rally idea is to make tool that simplifies all kinds of testing of multiple
OpenStack clouds.
It should be for human and as well simple to integrated in CI/CD process.

Rally automates all testing process (managing testing systems / running
tests / storing results / working with results)

At this moment there are 3 major parts:
*) deployment - manages OpenStack deployments (creates or uses existing)
*) verify - manages fully tempest (installs/configurtion/running/parsing
output/storing results/working with results)
*) task - own rally testing framework that allows you to do all kinds of
testing functional/load/performance/scale/load/volume and others.

I can say that "rally verify" command that automates work with Tempest is
very popular. More details here:
https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/

Proposal to make the life better


Recently Yair Fried and Prasanth Anbalagan proposed a great idea to extend
"rally verify" command to add ability to run in-tree functional tests in
the same way as tempest.

In other words to have next syntax: "rally verify "

Something like this:

rally verify swift start # 1. Check is installed swift for active rally
deployment.
# IF NO:
# Downloads from default (our
specified place) swift
# Switch to master or specified tag
# Installs in venv swift
# Configure swift functional test
config for active deployment
# 2. Run swift functional test
# 3. Parse subunit output and store to
Rally DB (for future work)

rally verify swift list # List all swift
verification runs
rally verify swift show UUID # Shows results
rally verify swift compare UUID1 UUID2 # Compare results of two runs

Why it makes sense?


1) Unification of testing process.

There is a simple to learn set of command "rally verify "
that works for all projects in the same way. End users like such things=)

2) Simplification of testing process.

"rally verify start" - will automate all steps, so you won't need
to install project manually, and configure functional test, collect and
somewhere store results.

3) Avoiding duplication of effort

We don't need to implement part of "rally verify" functionality in every
project.
It is better to implement it in one place with plugin support. Adding new
project means implementing new plugin (in most case it will be just
functional test conf generation)

4) Reusing already existing code

Most of the code that we need is already implemented in Rally,
it requires just small refactoring and generalization.

Thoughts?

Best regards,
Boris Pavlovic


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Mar 6, 2015 in openstack-dev by boris_at_pavlovic.me (6,900 points)   1 4 8
retagged Mar 19, 2015 by admin

7 Responses

0 votes

Davanum,

  1. Suppose a project say Nova wants to enable this Rally Integration
    > for its functional tests, what does that project have to do? (other
    > than the existing well defined tox targets)

Actually project by it self shouldn't do anything.

The whole work regarding to integration belongs to Infra and Rally code.
In Infra we should add job that runs few commands like:

rally deployment use devstack # this is already predefined in rally
devstack plugin
rally verify nova start # ...
rally verify nova results --html /files-that-will-be-plubished/result.html
rally verify nova results --json /files-that-will-be-plubished/result.json
rally verify nova check_status # return 1 if some of tests
failed

In Rally we should add auto generation on nova .conf file, based on
OpenStack credentials.

The whole idea is to make common, short and simple set of command that is
easy to integrate in gates and use locally.

2 more things:
1) We are going to support running specific releases of Rally in gates
(so projects won't depend on Rally master)
2) Rally has own DB where it stores full results of rally verify runs, if
we share this DB between different runs, we can collect all results of all
functional test runs in one place, and in future analyze (like generating
graphs failuresoftestxper/week)

  1. Is there a "test" project with Gabbi based tests that you know of?

As far as I know Ceilometer is first path-finder here:
https://github.com/openstack/ceilometer/tree/master/ceilometer/tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?

More or less no changes. It can be run via testr/nose that is enough.

Best regards,
Boris Pavlovic

On Mon, Mar 9, 2015 at 5:05 PM, Davanum Srinivas davanum@gmail.com wrote:

Boris,

  1. Suppose a project say Nova wants to enable this Rally Integration
    for its functional tests, what does that project have to do? (other
    than the existing well defined tox targets)
  2. Is there a "test" project with Gabbi based tests that you know of?
  3. What changes if any are needed in Gabbi to make this happen?

Guessing going forward, we can setup weekly Rally jobs against
different projects so we can compare performance over time etc?

thanks,
dims

On Fri, Mar 6, 2015 at 6:47 PM, Boris Pavlovic boris@pavlovic.me wrote:

Hi stackers,

Intro (Gabbi)


Gabbi is amazing tool that allows you to describe in human readable way
what
API requests to execute and what you are expecting as a result. It
Simplifies a lot API testing.

It's based on unittest so it can be easily run using tox/tester/nose and
so
on..

Intro (Functional in-tree tests)


Keeping all tests in one project like Tempest, that is maintained by one
team, was not enough scalable approach. To scale things, projects started
maintaining their own functional tests in their own tree. This resolves
scale issues and now new features can be merged with functional tests.

The Problem


As far as you know there are a lot of OpenStack projects with their own
functional tests / gabbi tests in tree. It becomes hard for developers,
devops and operators to work with them. (Like it's hard to install
OpenStack
by hands without DevStack. )

Usually, end users are choosing 2 approach:
1) Make own tests
2) Make scripts that runs somehow all these tests

Small Intro (Rally)


Rally idea is to make tool that simplifies all kinds of testing of
multiple
OpenStack clouds.
It should be for human and as well simple to integrated in CI/CD process.

Rally automates all testing process (managing testing systems / running
tests / storing results / working with results)

At this moment there are 3 major parts:
*) deployment - manages OpenStack deployments (creates or uses existing)
*) verify - manages fully tempest (installs/configurtion/running/parsing
output/storing results/working with results)
*) task - own rally testing framework that allows you to do all kinds of
testing functional/load/performance/scale/load/volume and others.

I can say that "rally verify" command that automates work with Tempest is
very popular. More details here:

https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/

Proposal to make the life better


Recently Yair Fried and Prasanth Anbalagan proposed a great idea to
extend
"rally verify" command to add ability to run in-tree functional tests in
the
same way as tempest.

In other words to have next syntax: "rally verify "

Something like this:

rally verify swift start # 1. Check is installed swift for active
rally
deployment.
# IF NO:
# Downloads from default (our
specified place) swift
# Switch to master or specified
tag
# Installs in venv swift
# Configure swift functional test
config for active deployment
# 2. Run swift functional test
# 3. Parse subunit output and store
to
Rally DB (for future work)

rally verify swift list # List all
swift
verification runs
rally verify swift show UUID # Shows results
rally verify swift compare UUID1 UUID2 # Compare results of two runs

Why it makes sense?


1) Unification of testing process.

There is a simple to learn set of command "rally verify "
that works for all projects in the same way. End users like such
things=)

2) Simplification of testing process.

"rally verify start" - will automate all steps, so you won't
need
to install project manually, and configure functional test, collect and
somewhere store results.

3) Avoiding duplication of effort

We don't need to implement part of "rally verify" functionality in every
project.
It is better to implement it in one place with plugin support. Adding new
project means implementing new plugin (in most case it will be just
functional test conf generation)

4) Reusing already existing code

Most of the code that we need is already implemented in Rally,
it requires just small refactoring and generalization.

Thoughts?

Best regards,
Boris Pavlovic


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Mar 9, 2015 by boris_at_pavlovic.me (6,900 points)   1 4 8
0 votes

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/tests/gabbi
  1. What changes if any are needed in Gabbi to make this happen?

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Mar 9, 2015 by Chris_Dent (7,940 points)   1 4 7
0 votes

Hi,

I like this idea, we use Rally for OpenStack clouds verification at scale
and it is the real issue - how to run all functional tests from each
project with the one script. If Rally will do this, I will use Rally to run
these tests.

Thank you!

On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent chdent@redhat.com wrote:

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?
    >

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?
    >

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 10, 2015 by Timur_Nurlygayanov (2,440 points)   1 4 8
0 votes

The idea is great, but IMHO we can move all project-specific code out of
rally, so:

  • rally plugin should be a part of project (for example, located in
    functional tests directory)
  • use {project url} instead of {project name} in rally verify command,
    example:

    $ rally verify https://github.com/openstack/nova start

On Tue, Mar 10, 2015 at 6:01 PM, Timur Nurlygayanov <
tnurlygayanov@mirantis.com> wrote:

Hi,

I like this idea, we use Rally for OpenStack clouds verification at scale
and it is the real issue - how to run all functional tests from each
project with the one script. If Rally will do this, I will use Rally to run
these tests.

Thank you!

On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent chdent@redhat.com wrote:

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?
    >

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?
    >

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 11, 2015 by Aleksandr_Maretskiy (640 points)   1
0 votes

$ rally verify https://github.com/openstack/nova start

As one of end-users of Rally, I dislike such construction, because I don't
want to remember links to repos, they are too long for me:)

On Wed, Mar 11, 2015 at 12:49 PM, Aleksandr Maretskiy <
amaretskiy@mirantis.com> wrote:

The idea is great, but IMHO we can move all project-specific code out of
rally, so:

  • rally plugin should be a part of project (for example, located in
    functional tests directory)
  • use {project url} instead of {project name} in rally verify command,
    example:

    $ rally verify https://github.com/openstack/nova start

On Tue, Mar 10, 2015 at 6:01 PM, Timur Nurlygayanov <
tnurlygayanov@mirantis.com> wrote:

Hi,

I like this idea, we use Rally for OpenStack clouds verification at scale
and it is the real issue - how to run all functional tests from each
project with the one script. If Rally will do this, I will use Rally to run
these tests.

Thank you!

On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent chdent@redhat.com wrote:

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?
    >

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?
    >

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 11, 2015 by Andrey_Kurilin (3,820 points)   2 4
0 votes

Alex,

  • rally plugin should be a part of project (for example, located in
    functional tests directory)

There are 2 issues with such solution:

1) If rally didn't load plugin, command "rally verify " won't
exist
2) Putting some strange Rally plugin to source of other projects will be
quite complicated task.
I believe we should have at least POC before even asking for such
stuff.

  • use {project url} instead of {project name} in rally verify command,
    example:

I agree here with Andrey, it is bad UX. Forcing people to write every time
URLs is terrible.
They will build own tools on top of such solution.

What about "rally verify nova start --url <....>" where --url is optional
argument?
If "--url" is not specified default url is used.

Best regards,
Boris Pavlovic

On Wed, Mar 11, 2015 at 7:14 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

$ rally verify https://github.com/openstack/nova start

As one of end-users of Rally, I dislike such construction, because I don't
want to remember links to repos, they are too long for me:)

On Wed, Mar 11, 2015 at 12:49 PM, Aleksandr Maretskiy <
amaretskiy@mirantis.com> wrote:

The idea is great, but IMHO we can move all project-specific code out of
rally, so:

  • rally plugin should be a part of project (for example, located in
    functional tests directory)
  • use {project url} instead of {project name} in rally verify command,
    example:

    $ rally verify https://github.com/openstack/nova start

On Tue, Mar 10, 2015 at 6:01 PM, Timur Nurlygayanov <
tnurlygayanov@mirantis.com> wrote:

Hi,

I like this idea, we use Rally for OpenStack clouds verification at
scale and it is the real issue - how to run all functional tests from each
project with the one script. If Rally will do this, I will use Rally to run
these tests.

Thank you!

On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent chdent@redhat.com wrote:

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?
    >

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/
tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?
    >

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 12, 2015 by boris_at_pavlovic.me (6,900 points)   1 4 8
0 votes

I have an alternative solution for rally verify <project> start command.
What do you think about similar management stuff for verifiers as we have
for deployments?

It requires several changes:
- rally verifiers command: Implementation of new command, which will
manage(install, reinstall, remove, configure) verifiers(tempest,
project-specific functional tests, gabbi and etc), will allow users to have
several tempest verifiers with different configurations or branches and
easily switch between them.
- rally verify command: The original verification command will check
selected verifier and contain only verifier-specific sub-commands.
- db changes: we will need to store information about verifiers

On Thu, Mar 12, 2015 at 2:33 AM, Boris Pavlovic boris@pavlovic.me wrote:

Alex,

  • rally plugin should be a part of project (for example, located in
    > functional tests directory)

There are 2 issues with such solution:

1) If rally didn't load plugin, command "rally verify " won't
exist
2) Putting some strange Rally plugin to source of other projects will be
quite complicated task.
I believe we should have at least POC before even asking for such
stuff.

  • use {project url} instead of {project name} in rally verify command,
    > example:

I agree here with Andrey, it is bad UX. Forcing people to write every time
URLs is terrible.
They will build own tools on top of such solution.

What about "rally verify nova start --url <....>" where --url is optional
argument?
If "--url" is not specified default url is used.

Best regards,
Boris Pavlovic

On Wed, Mar 11, 2015 at 7:14 PM, Andrey Kurilin akurilin@mirantis.com
wrote:

$ rally verify https://github.com/openstack/nova start

As one of end-users of Rally, I dislike such construction, because I
don't want to remember links to repos, they are too long for me:)

On Wed, Mar 11, 2015 at 12:49 PM, Aleksandr Maretskiy <
amaretskiy@mirantis.com> wrote:

The idea is great, but IMHO we can move all project-specific code out of
rally, so:

  • rally plugin should be a part of project (for example, located in
    functional tests directory)
  • use {project url} instead of {project name} in rally verify command,
    example:

    $ rally verify https://github.com/openstack/nova start

On Tue, Mar 10, 2015 at 6:01 PM, Timur Nurlygayanov <
tnurlygayanov@mirantis.com> wrote:

Hi,

I like this idea, we use Rally for OpenStack clouds verification at
scale and it is the real issue - how to run all functional tests from each
project with the one script. If Rally will do this, I will use Rally to run
these tests.

Thank you!

On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent chdent@redhat.com wrote:

On Mon, 9 Mar 2015, Davanum Srinivas wrote:

  1. Is there a "test" project with Gabbi based tests that you know of?
    >

In addition to the ceilometer tests that Boris pointed out gnocchi
is using it as well:

https://github.com/stackforge/gnocchi/tree/master/gnocchi/
tests/gabbi

  1. What changes if any are needed in Gabbi to make this happen?
    >

I was unable to tell from the original what "this" is and how gabbi
is involved but the above link ought to be able to show you how
gabbi can be used. There's also the docs (which could do with some
improvement, so suggestions or pull requests welcome):

http://gabbi.readthedocs.org/en/latest/

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Andrey Kurilin.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 19, 2015 by Andrey_Kurilin (3,820 points)   2 4
...