settingsLogin | Registersettings

[openstack-dev] [qa] Status of account creds in the [identity] section of tempest.conf

0 votes

We had a discussion about this at the qa meeting today around the
following proposal:

tl;dr The test accounts feature provides the same functionality as the
embedded credentials. We should deprecate the account information
embedded directly in tempest.conf in favor of test-accounts, and remove
those options at the beginning of the M cycle. We would also rework the
non-isolated jobs to use parallel test accounts, with and without admin
creds. Starting now, new features such as cleanup and tempest config
will not be required to work well (or at all) if the embedded creds are
used instead of test accounts.

We have (at least) three use cases that are important, and we want
tempest to work well with all of them, but that means something
different in each case:

  1. throw-away clouds (ci, gate)
  2. test clouds
  3. production clouds

For (1), the most important thing is that failing tests not cause false
negatives in other tests due to re-using a tenant. This makes tenant
isolation continue to be a good choice here, and requiring admin is not
an issue. In a perfect world where tempest never left behind any
resources regardless of an error at any line of code, test accounts
could be used. But we are probably a long way from that.

For (3), we cannot use admin creds for tempest runs, and test accounts
with cleanup allow parallel execution, accepting the risk of a leak
causing a false negative. The only way to avoid that risk is to stamp
out all leak bugs in tempest.

For (2), either isolation or test accounts with cleanup can be used

The tempest.conf values are not used in any of these scenarios. Is there
a reason they are needed for anything?

-David


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 18, 2015 in openstack-dev by David_Kranz (4,560 points)   1 3 3

2 Responses

0 votes

On Thu, Jun 18, 2015 at 02:13:56PM -0400, David Kranz wrote:
We had a discussion about this at the qa meeting today around the following
proposal:

tl;dr The test accounts feature provides the same functionality as the
embedded credentials. We should deprecate the account information embedded
directly in tempest.conf in favor of test-accounts, and remove those options
at the beginning of the M cycle. We would also rework the non-isolated jobs
to use parallel test accounts, with and without admin creds. Starting now,
new features such as cleanup and tempest config will not be required to work
well (or at all) if the embedded creds are used instead of test accounts.

So this was always the long term plan when we started work on the test-accounts
mechanism about a year ago. We were holding off on deprecating the original
config option based approach until finished the role and network support for
test accounts and we had jobs setup using the mechanism. Now that the we
finished both role and network support all that's left is setting up the jobs.
I don't think there would be any opposition to marking the user and alt user
options as deprecated after that. Also leaving in line comments (and maybe emit
a warning) marking the non-locking provider mechanism as going away, probably
in M. That way we start clearly marking to users that these options will be
going away.

We have (at least) three use cases that are important, and we want tempest
to work well with all of them, but that means something different in each
case:

  1. throw-away clouds (ci, gate)
  2. test clouds
  3. production clouds

Well, tempest is designed to and tries to support running against any OpenStack
cloud. I'm not sure if there are more deployment types than these 3 categories
but we should also be supporting those too.

For (1), the most important thing is that failing tests not cause false
negatives in other tests due to re-using a tenant. This makes tenant
isolation continue to be a good choice here, and requiring admin is not an
issue. In a perfect world where tempest never left behind any resources
regardless of an error at any line of code, test accounts could be used. But
we are probably a long way from that.

So the cleanup issue here is actually a wider openstack issue. Tempest will
always call delete on created projects and users. This was a requirement for
making test accounts work. (the mechanism for calling delete or freeing a
credential set from the list is shared) With tenant isolation this means we'll
be deleting a project and users and resources scoped to either may not be
deleted first. (if there is a tempest or openstack bug) This is a wider issue
for all OpenStack projects that there was a thread a few months ago discussing.

For (3), we cannot use admin creds for tempest runs, and test accounts with
cleanup allow parallel execution, accepting the risk of a leak causing a
false negative. The only way to avoid that risk is to stamp out all leak
bugs in tempest.

Well depending on the resource in leak in question test accounts would likely
catch the issues if there is a list on that resource in a later test. But, I
agree, resource leaks have always been treated as bugs and we'll continue to
do so.

For (2), either isolation or test accounts with cleanup can be used

The tempest.conf values are not used in any of these scenarios. Is there a
reason they are needed for anything?

So the only thing that uses config options for credentials is actually tenant
isolation, which uses them to provide admin credentials to do the dynamic
creation of accounts. The real advantage of tenant isolation, besides not
reusing anything, is actually its configuration simplicity. Using an accounts
file can be tricky to use, there are a lot of little gotchas and assumptions
in how you write the file. (which we try to document in both the config guide
and the sample accounts.yaml file) It also requires a large number of users to
be provided depending on the concurrency you're running with. While tenant
isolation requires just setting 3-5 config options and you're fine after that.

I don't think requiring the use of an accounts file for tenant isolation makes
much sense, it's really heavyweight for 1 set of admin creds. Which probably
means we should keep the admin user config option around. Although it might make
more sense to move those options to the auth section. (and maybe rename them to
make it clear that it's for tenant isolation only)

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jun 19, 2015 by Matthew_Treinish (11,200 points)   2 5 6
0 votes

On Fri, Jun 19, 2015 at 10:30 AM Matthew Treinish mtreinish@kortar.org
wrote:

On Thu, Jun 18, 2015 at 02:13:56PM -0400, David Kranz wrote:

We had a discussion about this at the qa meeting today around the
following
proposal:

tl;dr The test accounts feature provides the same functionality as the
embedded credentials. We should deprecate the account information
embedded
directly in tempest.conf in favor of test-accounts, and remove those
options
at the beginning of the M cycle. We would also rework the non-isolated
jobs
to use parallel test accounts, with and without admin creds.

+1

Starting now,

new features such as cleanup and tempest config will not be required to
work
well (or at all) if the embedded creds are used instead of test accounts.

So this was always the long term plan when we started work on the
test-accounts
mechanism about a year ago. We were holding off on deprecating the original
config option based approach until finished the role and network support
for
test accounts and we had jobs setup using the mechanism. Now that the we
finished both role and network support all that's left is setting up the
jobs.
I don't think there would be any opposition to marking the user and alt
user
options as deprecated after that. Also leaving in line comments (and maybe
emit
a warning) marking the non-locking provider mechanism as going away,
probably
in M. That way we start clearly marking to users that these options will be
going away.

+1 because the options are completely moving out of conf, we cannot use the
deprecation mechanism from oslo.config - emitting a warning is a good idea

We have (at least) three use cases that are important, and we want
tempest
to work well with all of them, but that means something different in each
case:

  1. throw-away clouds (ci, gate)
  2. test clouds
  3. production clouds

Well, tempest is designed to and tries to support running against any
OpenStack
cloud. I'm not sure if there are more deployment types than these 3
categories
but we should also be supporting those too.

For (1), the most important thing is that failing tests not cause false
negatives in other tests due to re-using a tenant. This makes tenant
isolation continue to be a good choice here, and requiring admin is not
an
issue. In a perfect world where tempest never left behind any resources
regardless of an error at any line of code, test accounts could be used.
But
we are probably a long way from that.

So the cleanup issue here is actually a wider openstack issue. Tempest will
always call delete on created projects and users. This was a requirement
for
making test accounts work. (the mechanism for calling delete or freeing a
credential set from the list is shared) With tenant isolation this means
we'll
be deleting a project and users and resources scoped to either may not be
deleted first. (if there is a tempest or openstack bug) This is a wider
issue
for all OpenStack projects that there was a thread a few months ago
discussing.

For (3), we cannot use admin creds for tempest runs, and test accounts
with
cleanup allow parallel execution, accepting the risk of a leak causing a
false negative. The only way to avoid that risk is to stamp out all leak
bugs in tempest.

Well depending on the resource in leak in question test accounts would
likely
catch the issues if there is a list on that resource in a later test. But,
I
agree, resource leaks have always been treated as bugs and we'll continue
to
do so.

For (2), either isolation or test accounts with cleanup can be used

The tempest.conf values are not used in any of these scenarios. Is there
a
reason they are needed for anything?

So the only thing that uses config options for credentials is actually
tenant
isolation, which uses them to provide admin credentials to do the dynamic
creation of accounts. The real advantage of tenant isolation, besides not
reusing anything, is actually its configuration simplicity. Using an
accounts
file can be tricky to use, there are a lot of little gotchas and
assumptions
in how you write the file. (which we try to document in both the config
guide
and the sample accounts.yaml file) It also requires a large number of
users to
be provided depending on the concurrency you're running with. While tenant
isolation requires just setting 3-5 config options and you're fine after
that.

I don't think requiring the use of an accounts file for tenant isolation
makes
much sense, it's really heavyweight for 1 set of admin creds. Which
probably
means we should keep the admin user config option around. Although it
might make
more sense to move those options to the auth section. (and maybe rename
them to
make it clear that it's for tenant isolation only)

I would also keep the admin account in tempest.conf.
And +1 to renaming the config options.
If tenant isolation is disabled, we will still need admin accounts for
several tests - but those will have to be provided in the YAML file,
otherwise we won't be able to support parallel run of admin tests.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 16, 2015 by Andrea_Frittoli (5,920 points)   1 2 3
...