settingsLogin | Registersettings

[openstack-dev] [OpenStack-Ansible] Splitting out generic testing components

0 votes

TL;DR We're doing a PoC to split out common testing plays/vars into a
generic repository, for the purposes of making in-role testing more uniform
and generic. Looking for feedback, comments, informal reviews, ideas etc!
https://github.com/andymcc/openstack-ansible-testing-core (Includes a
simple README)

We now have a lot of duplication after moving to a single role per
repository structure with specific testing per role. For example, almost
all repositories require Galera/Rabbit/Keystone in order to deploy testing
successfully. This has led to a scenario where each repository essentially
carries the same deployment code.

Aims:
- The primary aim of extracting the testing infrastructure into a single
repository is to reduce the cases where a simple change is needed, which
dominoes, causing a patch to each of 15 repositories. Instead, a change to
the uniform testing repository would resolve the issue for all other
repository's testing.
- In the future, we are looking to deploy scenario testing. For example, we
may want to test glance with a swift backend, or keystone with memcache. If
the testing play to deploy swift is already in a uniform repository, the
changes required to get glance testing enabled for that use case would be
minimal.
- This allows new projects to consume existing structure/playbooks to
deploy common components and vars, which should simplify the manner in
which new openstack-ansible projects set up their testing.

Steps taken so far:
- The deployment plays for each project have been split out into the
separate testing role.
- Each role only carries a specific "Test project" play.
- The test playbooks have been made generic so it uses the inventory
specified per repository (defining what hosts/roles there are).
- The test-vars have been put in the testing-repository and moved out of
the generic role.
- An override file has been created for each project and included using
"-e" (the highest precedence) but at present, of the 4 projects converted
the maximum number of overrides used is 2, so these overrides are minimal.
- Adjustments to tox.ini and var files references have been made to use the
new repository.

Further work to look into:
*- *It may be worth looking into making the tox.ini more generic, if we
were to make a sweeping change that impacts on tox.ini we would still need
to do changes to each repository. (I am not sure on how feasible this is
though)

Raised concerns:
- This creates a situation where a change may require me to make a change
to the central testing repository before changing the role repository. For
example, in order to get the generic testing for a keystone change I would
have to change the testing repository in advance, and then change the
keystone repository. Or override the var, adjust the testing repo and then
remove the keystone override.
- Changes to the testing-repo can cause all other repo tests (aside from
the overarching openstack-ansible repository) from breaking.

Where to find the PoC, what next?

The repository can be found here:
https://github.com/andymcc/openstack-ansible-testing-core

This is a proof of concept so in no way is it considered complete or
perfect, we are asking for more eyes on this; test it, let us know what you
like/do not like/want changed/additional ideas to improve.

Thanks!

Andy
irc: andymccr


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 3, 2016 in openstack-dev by Andy_McCrae (1,720 points)   2
retagged Jan 25, 2017 by admin

2 Responses

0 votes

Nice work Andy. I'm strongly in favor of getting the central test repo into Gerrit for broader reviews and once done, getting one or two patches up with roles making use of it.

  • Travis Truman

From: Andy McCrae andy.mccrae@gmail.com
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Friday, June 3, 2016 at 11:41 AM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: [openstack-dev] [OpenStack-Ansible] Splitting out generic testing components

TL;DR We're doing a PoC to split out common testing plays/vars into a generic repository, for the purposes of making in-role testing more uniform and generic. Looking for feedback, comments, informal reviews, ideas etc! https://github.com/andymcc/openstack-ansible-testing-core (Includes a simple README)

We now have a lot of duplication after moving to a single role per repository structure with specific testing per role. For example, almost all repositories require Galera/Rabbit/Keystone in order to deploy testing successfully. This has led to a scenario where each repository essentially carries the same deployment code.

Aims:
- The primary aim of extracting the testing infrastructure into a single repository is to reduce the cases where a simple change is needed, which dominoes, causing a patch to each of 15 repositories. Instead, a change to the uniform testing repository would resolve the issue for all other repository's testing.
- In the future, we are looking to deploy scenario testing. For example, we may want to test glance with a swift backend, or keystone with memcache. If the testing play to deploy swift is already in a uniform repository, the changes required to get glance testing enabled for that use case would be minimal.
- This allows new projects to consume existing structure/playbooks to deploy common components and vars, which should simplify the manner in which new openstack-ansible projects set up their testing.

Steps taken so far:
- The deployment plays for each project have been split out into the separate testing role.
- Each role only carries a specific "Test project" play.
- The test playbooks have been made generic so it uses the inventory specified per repository (defining what hosts/roles there are).
- The test-vars have been put in the testing-repository and moved out of the generic role.
- An override file has been created for each project and included using "-e" (the highest precedence) but at present, of the 4 projects converted the maximum number of overrides used is 2, so these overrides are minimal.
- Adjustments to tox.ini and var files references have been made to use the new repository.

Further work to look into:
- It may be worth looking into making the tox.ini more generic, if we were to make a sweeping change that impacts on tox.ini we would still need to do changes to each repository. (I am not sure on how feasible this is though)

Raised concerns:
- This creates a situation where a change may require me to make a change to the central testing repository before changing the role repository. For example, in order to get the generic testing for a keystone change I would have to change the testing repository in advance, and then change the keystone repository. Or override the var, adjust the testing repo and then remove the keystone override.
- Changes to the testing-repo can cause all other repo tests (aside from the overarching openstack-ansible repository) from breaking.

Where to find the PoC, what next?

The repository can be found here: https://github.com/andymcc/openstack-ansible-testing-core

This is a proof of concept so in no way is it considered complete or perfect, we are asking for more eyes on this; test it, let us know what you like/do not like/want changed/additional ideas to improve.

Thanks!

Andy
irc: andymccr


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 8, 2016 by Travis_Truman_at_com (320 points)   2 2
0 votes

+1 I think this is a great initiative and should go a long way to
unifying and improving our tests within the roles. I too would love to
see this moved to the namespace so that folks can begin working on and
reviewing changes gerrit.

@Jesse, assuming everything is on the up and up, what are the next
steps to getting this setup in our project-config?

--

Kevin Carter
IRC: cloudnull

On Fri, Jul 8, 2016 at 9:18 AM, Truman, Travis
Travis_Truman@comcast.com wrote:
Nice work Andy. I’m strongly in favor of getting the central test repo into
Gerrit for broader reviews and once done, getting one or two patches up with
roles making use of it.

  • Travis Truman

From: Andy McCrae andy.mccrae@gmail.com
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Date: Friday, June 3, 2016 at 11:41 AM
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: [openstack-dev] [OpenStack-Ansible] Splitting out generic testing
components

TL;DR We're doing a PoC to split out common testing plays/vars into a
generic repository, for the purposes of making in-role testing more uniform
and generic. Looking for feedback, comments, informal reviews, ideas etc!
https://github.com/andymcc/openstack-ansible-testing-core (Includes a simple
README)

We now have a lot of duplication after moving to a single role per
repository structure with specific testing per role. For example, almost all
repositories require Galera/Rabbit/Keystone in order to deploy testing
successfully. This has led to a scenario where each repository essentially
carries the same deployment code.

Aims:
- The primary aim of extracting the testing infrastructure into a single
repository is to reduce the cases where a simple change is needed, which
dominoes, causing a patch to each of 15 repositories. Instead, a change to
the uniform testing repository would resolve the issue for all other
repository's testing.
- In the future, we are looking to deploy scenario testing. For example, we
may want to test glance with a swift backend, or keystone with memcache. If
the testing play to deploy swift is already in a uniform repository, the
changes required to get glance testing enabled for that use case would be
minimal.
- This allows new projects to consume existing structure/playbooks to deploy
common components and vars, which should simplify the manner in which new
openstack-ansible projects set up their testing.

Steps taken so far:
- The deployment plays for each project have been split out into the
separate testing role.
- Each role only carries a specific "Test project" play.
- The test playbooks have been made generic so it uses the inventory
specified per repository (defining what hosts/roles there are).
- The test-vars have been put in the testing-repository and moved out of the
generic role.
- An override file has been created for each project and included using "-e"
(the highest precedence) but at present, of the 4 projects converted the
maximum number of overrides used is 2, so these overrides are minimal.
- Adjustments to tox.ini and var files references have been made to use the
new repository.

Further work to look into:
- It may be worth looking into making the tox.ini more generic, if we were
to make a sweeping change that impacts on tox.ini we would still need to do
changes to each repository. (I am not sure on how feasible this is though)

Raised concerns:
- This creates a situation where a change may require me to make a change to
the central testing repository before changing the role repository. For
example, in order to get the generic testing for a keystone change I would
have to change the testing repository in advance, and then change the
keystone repository. Or override the var, adjust the testing repo and then
remove the keystone override.
- Changes to the testing-repo can cause all other repo tests (aside from the
overarching openstack-ansible repository) from breaking.

Where to find the PoC, what next?

The repository can be found here:
https://github.com/andymcc/openstack-ansible-testing-core

This is a proof of concept so in no way is it considered complete or
perfect, we are asking for more eyes on this; test it, let us know what you
like/do not like/want changed/additional ideas to improve.

Thanks!

Andy
irc: andymccr


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 12, 2016 by Carter,_Kevin (580 points)   1
...