settingsLogin | Registersettings

[openstack-dev] [keystone][nova] Persistent application credentials

0 votes

So the application credentials spec has merged - huge thanks to Monty
and the Keystone team for getting this done:

https://review.openstack.org/#/c/450415/
http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html

However, it appears that there was a disconnect in how two groups of
folks were reading the spec that only became apparent towards the end of
the process. Specifically, at this exact moment:

http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59

To summarise, Keystone folks are uncomfortable with the idea of
application credentials that share the lifecycle of the project (rather
than the user that created them), because a consumer could
surreptitiously create an application credential and continue to use
that to access the OpenStack APIs even after their User account is
deleted. The agreed solution was to delete the application credentials
when the User that created them is deleted, thus tying the lifecycle to
that of the User.

This means that teams using this feature will need to audit all of their
applications for credential usage and rotate any credentials created by
a soon-to-be-former team member before removing said team member's
User account, or risk breakage. Basically we're relying on users to do
the Right Thing (bad), but when they don't we're defaulting to breaking
[some of] their apps over leaving them insecure (all things being equal,
good).

Unfortunately, if we do regard this as a serious problem, I don't think
this solution is sufficient. Assuming that application credentials are
stored on VMs in the project for use by the applications running on
them, then anyone with access to those servers can obtain the
credentials and continue to use them even if their own account is
deleted. The solution to this is to rotate all application keys when a
user is deleted. So really we're relying on users to do the Right Thing
(bad), but when they don't we're defaulting to breaking [some of] their
apps and [potentially] leaving them insecure (worst possible combination).

(We're also being inconsistent, because according to the spec if you
revoke a role from a User then any application credentials they've
created that rely on that role continue to work. It's only if you delete
the User that they're revoked.)

As far as I can see, there are only two solutions to the fundamental
problem:

1) Fine-grained user-defined access control. We can minimise the set of
things that the application credentials are authorised to do. That's out
of scope for this spec, but something we're already planning as a future
enhancement.
2) Automated regular rotation of credentials. We can make sure that
whatever a departing team member does manage to hang onto quickly
becomes useless.

By way of comparison, AWS does both. There's fine-grained defined access
control in the form of IAM Roles, and these Roles can be associated with
EC2 servers. The servers have an account with rotating keys provided
through the metadata server. I can't find the exact period of rotation
documented, but it's on the order of magnitude of 1 hour.

There's plenty not to like about this design. Specifically, it's 2017
not 2007 and the idea that there's no point offering to segment
permissions at a finer grained level than that of a VM no longer holds
water IMHO, thanks to SELinux and containers. It'd be nice to be able to
provide multiple sets of credentials to different services running on a
VM, and it's probably essential to our survival that we find a way to
provide individual credentials to containers. Nevertheless, what they
have does solve the problem.

Note that there's pretty much no sane way for the user to automate
credential rotation themselves, because it's turtles all the way down.
e.g. it's easy in principle to set up a Heat template with a Mistral
workflow that will rotate the credentials for you, but they'll do so
using trusts that are, in turn, tied back to the consumer who created
the stack. (It suddenly occurs to me that this is a problem that all
services using trusts are going to need to solve.) Somewhere it all has
to be tied back to something that survives the entire lifecycle of the
project.

Would Keystone folks be happy to allow persistent credentials once we
have a way to hand out only the minimum required privileges?

If not I think we're back to https://review.openstack.org/#/c/222293/

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 30, 2017 in openstack-dev by Zane_Bitter (21,640 points)   3 5 9

18 Responses

0 votes

On Mon, Jul 17, 2017 at 6:39 PM, Zane Bitter zbitter@redhat.com wrote:

So the application credentials spec has merged - huge thanks to Monty and
the Keystone team for getting this done:

https://review.openstack.org/#/c/450415/
http://specs.openstack.org/openstack/keystone-specs/specs/
keystone/pike/application-credentials.html

However, it appears that there was a disconnect in how two groups of folks
were reading the spec that only became apparent towards the end of the
process. Specifically, at this exact moment:

http://eavesdrop.openstack.org/irclogs/%23openstack-keystone
/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59

To summarise, Keystone folks are uncomfortable with the idea of
application credentials that share the lifecycle of the project (rather
than the user that created them), because a consumer could surreptitiously
create an application credential and continue to use that to access the
OpenStack APIs even after their User account is deleted. The agreed
solution was to delete the application credentials when the User that
created them is deleted, thus tying the lifecycle to that of the User.

This means that teams using this feature will need to audit all of their
applications for credential usage and rotate any credentials created by a
soon-to-be-former team member before removing said team member's User
account, or risk breakage. Basically we're relying on users to do the Right
Thing (bad), but when they don't we're defaulting to breaking [some of]
their apps over leaving them insecure (all things being equal, good).

Unfortunately, if we do regard this as a serious problem, I don't think
this solution is sufficient. Assuming that application credentials are
stored on VMs in the project for use by the applications running on them,
then anyone with access to those servers can obtain the credentials and
continue to use them even if their own account is deleted. The solution to
this is to rotate all application keys when a user is deleted. So really
we're relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps and [potentially]
leaving them insecure (worst possible combination).

(We're also being inconsistent, because according to the spec if you
revoke a role from a User then any application credentials they've created
that rely on that role continue to work. It's only if you delete the User
that they're revoked.)

As far as I can see, there are only two solutions to the fundamental
problem:

1) Fine-grained user-defined access control. We can minimise the set of
things that the application credentials are authorised to do. That's out of
scope for this spec, but something we're already planning as a future
enhancement.
2) Automated regular rotation of credentials. We can make sure that
whatever a departing team member does manage to hang onto quickly becomes
useless.

By way of comparison, AWS does both. There's fine-grained defined access
control in the form of IAM Roles, and these Roles can be associated with
EC2 servers. The servers have an account with rotating keys provided
through the metadata server. I can't find the exact period of rotation
documented, but it's on the order of magnitude of 1 hour.

There's plenty not to like about this design. Specifically, it's 2017 not
2007 and the idea that there's no point offering to segment permissions at
a finer grained level than that of a VM no longer holds water IMHO, thanks
to SELinux and containers. It'd be nice to be able to provide multiple sets
of credentials to different services running on a VM, and it's probably
essential to our survival that we find a way to provide individual
credentials to containers. Nevertheless, what they have does solve the
problem.

Note that there's pretty much no sane way for the user to automate
credential rotation themselves, because it's turtles all the way down. e.g.
it's easy in principle to set up a Heat template with a Mistral workflow
that will rotate the credentials for you, but they'll do so using trusts
that are, in turn, tied back to the consumer who created the stack. (It
suddenly occurs to me that this is a problem that all services using trusts
are going to need to solve.) Somewhere it all has to be tied back to
something that survives the entire lifecycle of the project.

Would Keystone folks be happy to allow persistent credentials once we have
a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application credentials
dependent on several cycles of policy work. Right?

If not I think we're back to https://review.openstack.org/#/c/222293/

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 18, 2017 by Lance_Bragstad (11,080 points)   2 3 4
0 votes

On Tue, Jul 18, 2017 at 1:39 AM, Zane Bitter zbitter@redhat.com wrote:

So the application credentials spec has merged - huge thanks to Monty and
the Keystone team for getting this done:

https://review.openstack.org/#/c/450415/
http://specs.openstack.org/openstack/keystone-specs/specs/
keystone/pike/application-credentials.html

However, it appears that there was a disconnect in how two groups of folks
were reading the spec that only became apparent towards the end of the
process. Specifically, at this exact moment:

http://eavesdrop.openstack.org/irclogs/%23openstack-keystone
/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59

To summarise, Keystone folks are uncomfortable with the idea of
application credentials that share the lifecycle of the project (rather
than the user that created them), because a consumer could surreptitiously
create an application credential and continue to use that to access the
OpenStack APIs even after their User account is deleted. The agreed
solution was to delete the application credentials when the User that
created them is deleted, thus tying the lifecycle to that of the User.

This means that teams using this feature will need to audit all of their
applications for credential usage and rotate any credentials created by a
soon-to-be-former team member before removing said team member's User
account, or risk breakage. Basically we're relying on users to do the Right
Thing (bad), but when they don't we're defaulting to breaking [some of]
their apps over leaving them insecure (all things being equal, good).

Unfortunately, if we do regard this as a serious problem, I don't think
this solution is sufficient. Assuming that application credentials are
stored on VMs in the project for use by the applications running on them,
then anyone with access to those servers can obtain the credentials and
continue to use them even if their own account is deleted. The solution to
this is to rotate all application keys when a user is deleted. So really
we're relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps and [potentially]
leaving them insecure (worst possible combination).

(We're also being inconsistent, because according to the spec if you
revoke a role from a User then any application credentials they've created
that rely on that role continue to work. It's only if you delete the User
that they're revoked.)

As far as I can see, there are only two solutions to the fundamental
problem:

1) Fine-grained user-defined access control. We can minimise the set of
things that the application credentials are authorised to do. That's out of
scope for this spec, but something we're already planning as a future
enhancement.
2) Automated regular rotation of credentials. We can make sure that
whatever a departing team member does manage to hang onto quickly becomes
useless.

By way of comparison, AWS does both. There's fine-grained defined access
control in the form of IAM Roles, and these Roles can be associated with
EC2 servers. The servers have an account with rotating keys provided
through the metadata server. I can't find the exact period of rotation
documented, but it's on the order of magnitude of 1 hour.

There's plenty not to like about this design. Specifically, it's 2017 not
2007 and the idea that there's no point offering to segment permissions at
a finer grained level than that of a VM no longer holds water IMHO, thanks
to SELinux and containers. It'd be nice to be able to provide multiple sets
of credentials to different services running on a VM, and it's probably
essential to our survival that we find a way to provide individual
credentials to containers. Nevertheless, what they have does solve the
problem.

Note that there's pretty much no sane way for the user to automate
credential rotation themselves, because it's turtles all the way down. e.g.
it's easy in principle to set up a Heat template with a Mistral workflow
that will rotate the credentials for you, but they'll do so using trusts
that are, in turn, tied back to the consumer who created the stack. (It
suddenly occurs to me that this is a problem that all services using trusts
are going to need to solve.) Somewhere it all has to be tied back to
something that survives the entire lifecycle of the project.

Would Keystone folks be happy to allow persistent credentials once we have
a way to hand out only the minimum required privileges?

I agree that in the haste of getting this approved before the spec freeze
deadline we took this in the wrong direction. I think that this spec was
fine before the addition of "Will be deleted when the associated User is
deleted" constraint.

As I understand it, the worry coming from the team is that a user who
sneakily copies the secret keys to an offsite location before their user is
deleted will still have access to the resources the credential had access
to. This worry is irrational for two reasons:

1) Once the application credential is created, it is going to be shared
with the whole team. Therefore if ANY member of the team is deleted,
whether or not they were the one to create the credential, they could still
have copied this key and still gain access to those resources after their
user was terminated. Tying the lifecycle of the credential to its creating
user does nothing to prevent other teammates from regaining this access.

2) The ability to rotate the secret, whether it's automated or not, is
already an infinitely better situation than where we currently are, which
is that a user must live forever and share their personal passwords with
the whole team in order to keep applications running. We can encourage
immediate credential rotation upon user termination and provide helpful
tooling and perhaps Heat integration, but even without all of that it's
still better than forcing users to share their passwords with one another
and being unable to delete users who have left the organization.

I'm supportive of revising the spec to undo this constraint.

Colleen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 18, 2017 by colleen_at_gazlene.n (1,780 points)   2
0 votes

On 07/17/2017 10:12 PM, Lance Bragstad wrote:

On Mon, Jul 17, 2017 at 6:39 PM, Zane Bitter <zbitter@redhat.com
zbitter@redhat.com> wrote:

So the application credentials spec has merged - huge thanks to
Monty and the Keystone team for getting this done:

https://review.openstack.org/#/c/450415/
<https://review.openstack.org/#/c/450415/>
http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html


However, it appears that there was a disconnect in how two groups
of folks were reading the spec that only became apparent towards
the end of the process. Specifically, at this exact moment:

http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59


To summarise, Keystone folks are uncomfortable with the idea of
application credentials that share the lifecycle of the project
(rather than the user that created them), because a consumer could
surreptitiously create an application credential and continue to
use that to access the OpenStack APIs even after their User
account is deleted. The agreed solution was to delete the
application credentials when the User that created them is
deleted, thus tying the lifecycle to that of the User.

This means that teams using this feature will need to audit all of
their applications for credential usage and rotate any credentials
created by a soon-to-be-former team member *before* removing said
team member's User account, or risk breakage. Basically we're
relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps over leaving
them insecure (all things being equal, good).

Unfortunately, if we do regard this as a serious problem, I don't
think this solution is sufficient. Assuming that application
credentials are stored on VMs in the project for use by the
applications running on them, then anyone with access to those
servers can obtain the credentials and continue to use them even
if their own account is deleted. The solution to this is to rotate
*all* application keys when a user is deleted. So really we're
relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps *and*
[potentially] leaving them insecure (worst possible combination).

(We're also being inconsistent, because according to the spec if
you revoke a role from a User then any application credentials
they've created that rely on that role continue to work. It's only
if you delete the User that they're revoked.)


As far as I can see, there are only two solutions to the
fundamental problem:

1) Fine-grained user-defined access control. We can minimise the
set of things that the application credentials are authorised to
do. That's out of scope for this spec, but something we're already
planning as a future enhancement.
2) Automated regular rotation of credentials. We can make sure
that whatever a departing team member does manage to hang onto
quickly becomes useless.

By way of comparison, AWS does both. There's fine-grained defined
access control in the form of IAM Roles, and these Roles can be
associated with EC2 servers. The servers have an account with
rotating keys provided through the metadata server. I can't find
the exact period of rotation documented, but it's on the order of
magnitude of 1 hour.

There's plenty not to like about this design. Specifically, it's
2017 not 2007 and the idea that there's no point offering to
segment permissions at a finer grained level than that of a VM no
longer holds water IMHO, thanks to SELinux and containers. It'd be
nice to be able to provide multiple sets of credentials to
different services running on a VM, and it's probably essential to
our survival that we find a way to provide individual credentials
to containers. Nevertheless, what they have does solve the problem.

Note that there's pretty much no sane way for the user to automate
credential rotation themselves, because it's turtles all the way
down. e.g. it's easy in principle to set up a Heat template with a
Mistral workflow that will rotate the credentials for you, but
they'll do so using trusts that are, in turn, tied back to the
consumer who created the stack. (It suddenly occurs to me that
this is a problem that all services using trusts are going to need
to solve.) Somewhere it all has to be tied back to something that
survives the entire lifecycle of the project.

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application
credentials dependent on several cycles of policy work. Right?

I think having the ability to communicate deprecations though
oslo.policy would help here. We could use it to move towards better
default roles, which requires being able to set minimum privileges.

Using the current workflow requires operators to define the minimum
privileges for whatever is using the application credential, and work
that into their policy. Is that the intended workflow that we want to
put on the users and operators of application credentials?

If not I think we're back to
https://review.openstack.org/#/c/222293/
<https://review.openstack.org/#/c/222293/>

cheers,
Zane.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 18, 2017 by Lance_Bragstad (11,080 points)   2 3 4
0 votes

On 17/07/17 23:12, Lance Bragstad wrote:
Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application credentials
dependent on several cycles of policy work. Right?

My thought here was that if this were the case (i.e. persistent
credentials are OK provided the user can lock down the privileges) then
you could make a case that the current spec is on the right track. For
now we implement the application credentials as non-persistent, people
who know about it use at their own risk, and for people who don't
there's no exposure. Later on we add the authorisation stuff and relax
the non-persistence requirement.

On further reflection, I'm not convinced by this - if we care about
protecting people who don't intentionally use/know about the feature
now, then we should probably still care once the tools are in place for
the people who are using it intentionally to lock it down tightly.

So I'm increasingly convinced that we need to do one of two things. Either:

  • Agree with Colleen (elsewhere in the thread) that persistent
    application credentials are still better than the status quo and
    reinstate the project-scoped lifecycle in accordance with original
    intent of the spec; or

  • Agree that the concerns raised by Morgan & Adam will always apply, and
    look for a solution that gives us automatic key rotation - which might
    be quite different in shape (I can elaborate if necessary).

(That said, I chatted about this briefly with Monty yesterday and he
said that his recollection was that there is a long-term solution that
will keep everyone happy. He'll try to remember what it is once he's
finished on the version discovery stuff he's currently working on.)

I'm trying to avoid taking a side here because everyone is right.
Currently anybody who want to do anything remotely 'cloudy' (i.e. have
the application talk to OpenStack APIs) has to either share their
personal password with the application (and by extension their whole
team) or to do the thing that is the polar opposite of cloud: file a
ticket with IT to get a service user account added and share that password instead. And this really is a disaster for
OpenStack. On the other hand, allowing the creation of persistent
application credentials in the absence of regular automatic rotation
does create risk for those folks who are not aggressively auditing them
(perhaps because they have no legitimate use for them) and the result is
likely to be lots of clouds disabling them by policy, keeping their
users in the dark age of IT-ticket-filing <head...desk> and frustrating
our interoperability goals.

It is possible in theory to satisfy both via the 'instance users'
concept, but the Nova team's response to this has consistently been
"prove to us that this has to be in Nova". Well, here's your answer.

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 18, 2017 by Zane_Bitter (21,640 points)   3 5 9
0 votes

On 18/07/17 10:55, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application
credentials dependent on several cycles of policy work. Right?

I think having the ability to communicate deprecations though
oslo.policy would help here. We could use it to move towards better
default roles, which requires being able to set minimum privileges.

Using the current workflow requires operators to define the minimum
privileges for whatever is using the application credential, and work
that into their policy. Is that the intended workflow that we want to
put on the users and operators of application credentials?

The plan is to add an authorisation mechanism that is user-controlled
and independent of the (operator-controlled) policy. The beginnings of
this were included in earlier drafts of the spec, but were removed in
patch set 19 in favour of leaving them for a future spec:

https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst

  • ZB


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 18, 2017 by Zane_Bitter (21,640 points)   3 5 9
0 votes

On 19/07/17 01:23, Colleen Murphy wrote:
On Tue, Jul 18, 2017 at 1:39 AM, Zane Bitter <zbitter@redhat.com
zbitter@redhat.com> wrote:

So the application credentials spec has merged - huge thanks to
Monty and the Keystone team for getting this done:

https://review.openstack.org/#/c/450415/
<https://review.openstack.org/#/c/450415/>
http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html


However, it appears that there was a disconnect in how two groups
of folks were reading the spec that only became apparent towards
the end of the process. Specifically, at this exact moment:

http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59


To summarise, Keystone folks are uncomfortable with the idea of
application credentials that share the lifecycle of the project
(rather than the user that created them), because a consumer could
surreptitiously create an application credential and continue to
use that to access the OpenStack APIs even after their User
account is deleted. The agreed solution was to delete the
application credentials when the User that created them is
deleted, thus tying the lifecycle to that of the User.

This means that teams using this feature will need to audit all of
their applications for credential usage and rotate any credentials
created by a soon-to-be-former team member *before* removing said
team member's User account, or risk breakage. Basically we're
relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps over leaving
them insecure (all things being equal, good).

Unfortunately, if we do regard this as a serious problem, I don't
think this solution is sufficient. Assuming that application
credentials are stored on VMs in the project for use by the
applications running on them, then anyone with access to those
servers can obtain the credentials and continue to use them even
if their own account is deleted. The solution to this is to rotate
*all* application keys when a user is deleted. So really we're
relying on users to do the Right Thing (bad), but when they don't
we're defaulting to breaking [some of] their apps *and*
[potentially] leaving them insecure (worst possible combination).

(We're also being inconsistent, because according to the spec if
you revoke a role from a User then any application credentials
they've created that rely on that role continue to work. It's only
if you delete the User that they're revoked.)


As far as I can see, there are only two solutions to the
fundamental problem:

1) Fine-grained user-defined access control. We can minimise the
set of things that the application credentials are authorised to
do. That's out of scope for this spec, but something we're already
planning as a future enhancement.
2) Automated regular rotation of credentials. We can make sure
that whatever a departing team member does manage to hang onto
quickly becomes useless.

By way of comparison, AWS does both. There's fine-grained defined
access control in the form of IAM Roles, and these Roles can be
associated with EC2 servers. The servers have an account with
rotating keys provided through the metadata server. I can't find
the exact period of rotation documented, but it's on the order of
magnitude of 1 hour.

There's plenty not to like about this design. Specifically, it's
2017 not 2007 and the idea that there's no point offering to
segment permissions at a finer grained level than that of a VM no
longer holds water IMHO, thanks to SELinux and containers. It'd be
nice to be able to provide multiple sets of credentials to
different services running on a VM, and it's probably essential to
our survival that we find a way to provide individual credentials
to containers. Nevertheless, what they have does solve the problem.

Note that there's pretty much no sane way for the user to automate
credential rotation themselves, because it's turtles all the way
down. e.g. it's easy in principle to set up a Heat template with a
Mistral workflow that will rotate the credentials for you, but
they'll do so using trusts that are, in turn, tied back to the
consumer who created the stack. (It suddenly occurs to me that
this is a problem that all services using trusts are going to need
to solve.) Somewhere it all has to be tied back to something that
survives the entire lifecycle of the project.

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

I agree that in the haste of getting this approved before the spec
freeze deadline we took this in the wrong direction. I think that this
spec was fine before the addition of "Will be deleted when the
associated User is deleted" constraint.

As I understand it, the worry coming from the team is that a user who
sneakily copies the secret keys to an offsite location before their
user is deleted will still have access to the resources the credential
had access to. This worry is irrational for two reasons:

1) Once the application credential is created, it is going to be
shared with the whole team. Therefore if ANY member of the team is
deleted, whether or not they were the one to create the credential,
they could still have copied this key and still gain access to those
resources after their user was terminated. Tying the lifecycle of the
credential to its creating user does nothing to prevent other
teammates from regaining this access.

2) The ability to rotate the secret, whether it's automated or not, is
already an infinitely better situation than where we currently are,
which is that a user must live forever and share their personal
passwords with the whole team in order to keep applications running.
We can encourage immediate credential rotation upon user termination
and provide helpful tooling and perhaps Heat integration, but even
without all of that it's still better than forcing users to share
their passwords with one another and being unable to delete users who
have left the organization.

I'm supportive of revising the spec to undo this constraint.

While I understand worries behind these security problems, I'd like to
add they they aren't really our problem. They can't be solved with code
on our end. These are procedural problems, and this problem persists in
any organisation, in any form of shared user passwords, in any form of
automation. If an application credential is created for a service
used/maintained by a team, there is nothing stopping anyone from the
team from storing that key elsewhere regardless of who created it,
because once created, that key is in the 'shared' team domain where
anyone can see it, exactly as Colleen points out. If it wasn't shared
with more than one person, then it's useless since it means the service
using the key itself would entirely have to be maintained/controlled by
that one person and no one else would have access. Teams don't work like
that.

The problem is then entirely procedural within a team. Do they rotate
all keys when one person leaves? Anything less is the same problem. All
we can do is make rotation less of a pain, but it will still be painful
no matter what, and depending on the situation the team makes the choice
of how to handle rotation if at all.

The sole reason for project level ownership of these application
credentials is so that a user leaving/being deleted isn't a scramble to
replace keys, and a team has the option/time to do it if they care about
the possibility of that person having known the keys (again, not our
problem, not a security flaw in code). Anything else, pretty much makes
this feature useless for teams. :(

Having both options (owned by project vs user) is useful, but the
'security issues' are kind of implied by using project owned app creds.
It's a very useful feature with some 'use at your own risk' attached.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 20, 2017 by Adrian_Turjak (2,660 points)   3 6
0 votes

On 07/19/2017 12:18 AM, Zane Bitter wrote:
On 18/07/17 10:55, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application
credentials dependent on several cycles of policy work. Right?

I think having the ability to communicate deprecations though
oslo.policy would help here. We could use it to move towards better
default roles, which requires being able to set minimum privileges.

Using the current workflow requires operators to define the minimum
privileges for whatever is using the application credential, and work
that into their policy. Is that the intended workflow that we want to
put on the users and operators of application credentials?

The plan is to add an authorisation mechanism that is user-controlled
and independent of the (operator-controlled) policy. The beginnings of
this were included in earlier drafts of the spec, but were removed in
patch set 19 in favour of leaving them for a future spec:

https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst

Yes - that's right - and I expect to start work on that again as soon as
this next keystoneauth release with version discovery is out the door.

It turns out there are different POVs on this topic, and it's VERY
important to be clear which one we're talking about at any given point
in time. A bunch of the confusion just in getting as far as we've gotten
so far came from folks saying words like "policy" or "trusts" or "ACLs"
or "RBAC" - but not clarifying which group of cloud users they were
discussing and from what context.

The problem that Zane and I are are discussing and advocating for are
for UNPRIVILEDGED users who neither deploy nor operate the cloud but who
use the cloud to run applications.

Unfortunately, neither the current policy system nor trusts are useful
in any way shape or form for those humans. Policy and trusts are tools
for cloud operators to take a certain set of actions.

Similarly, the concern from the folks who are not in favor of
project-lifecycled application credentials is the one that Zane outlined
- that there will be $someone with access to those credentials after a
User change event, and thus $security will be compromised.

There is a balance that can and must be found. The use case Zane and I
are talking about is ESSENTIAL, and literally ever single human who is a
actually using OpenStack to run applications needs it. Needed it last
year in fact, and they are, in fact doing things like writing ssh-agent
like daemons in which they can store their corporate LDAP credentials so
that their automation will work because we're not giving them a workable
option.

That said, the concerns about not letting a thing out the door that is
insecure by design like PHP4's globally scoped URL variables is also
super important.

So we need to find a design that meets both goals.

I have thoughts on the topic, but have been holding off until
version-discovery is out the door. My hunch is that, like application
credentials, we're not going to make significant headway without getting
humans in the room - because the topic is WAY too fraught with peril.

I propose we set aside time at the PTG to dig in to this. Between Zane
and I and the Keystone core team I have confidence we can find a way out.

Monty

PS. It will not help to solve limited-scope before we solve this.
Limited scope is an end-user opt-in action and having it does not remove
the concerns that have been expressed.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 20, 2017 by Monty_Taylor (22,780 points)   2 4 7
0 votes

On 07/19/2017 12:11 AM, Zane Bitter wrote:
On 17/07/17 23:12, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application
credentials dependent on several cycles of policy work. Right?

My thought here was that if this were the case (i.e. persistent
credentials are OK provided the user can lock down the privileges) then
you could make a case that the current spec is on the right track. For
now we implement the application credentials as non-persistent, people
who know about it use at their own risk, and for people who don't
there's no exposure. Later on we add the authorisation stuff and relax
the non-persistence requirement.

On further reflection, I'm not convinced by this - if we care about
protecting people who don't intentionally use/know about the feature
now, then we should probably still care once the tools are in place for
the people who are using it intentionally to lock it down tightly.

So I'm increasingly convinced that we need to do one of two things. Either:

  • Agree with Colleen (elsewhere in the thread) that persistent
    application credentials are still better than the status quo and
    reinstate the project-scoped lifecycle in accordance with original
    intent of the spec; or

  • Agree that the concerns raised by Morgan & Adam will always apply, and
    look for a solution that gives us automatic key rotation - which might
    be quite different in shape (I can elaborate if necessary).

(That said, I chatted about this briefly with Monty yesterday and he
said that his recollection was that there is a long-term solution that
will keep everyone happy. He'll try to remember what it is once he's
finished on the version discovery stuff he's currently working on.)

Part of this comes down to the fact that there are actually multiple
scenarios and persistent credentials actually only applies to a scenario
that typically requires a human with elevated credentials.

SO - I think we can get a long way forward by divvying up some
responsibilities clearly.

What I mean is:

  • The simple consume case ("typical public cloud") is User-per-Project
    with User lifecycle tied to Project lifecycle. In this case the idea of
    a 'persistent' credential is meaningless, because there is no 'other'
    User with access to the Project If the User in this scenario creates a
    Credential it doesn't actually matter what the Credential lifecycle is
    because the act of Account is ultimately about disabling or deleting
    access to the Project in question. We can and should help the folks who
    are running clouds in this model with $something (we need to talk
    details) so that if they are running in this model they don't
    accidentally or by default leave a door open when they think they've
    disabled someone's User as part of shutting off their Account. But in
    this scenario OpenStack adding project-persistent credentials is not a
    big deal - it doesn't provide value. (While a User in that scenario, who
    typically does not have the Role to create a new User being able to
    manage Application Credentials is a HUGE win)

  • The other scenario is where there is more than one Human who has a
    User that have been granted Roles on a Project. This is the one where
    project-lifecycle credentials are meaningful and valuable, but it's also
    one that involves some Human with elevated admin-style privileges having
    been involved at some point because that is required to assign Users
    Roles in the Project in the first place.

I believe if we divide application credentials into two kinds:

1) Application Credentials with lifecycle tied to User
2) Application Credentials with lifecycle tied to Project

Then I think it's ok for the ability to do (2) to require a specific
Role in policy. If we do that, then whatever Human it is that is mapping
multiple Users into a single Project can decide whether any of those
Users should be granted the ability to make Project-lifecycle
Application Credentials. Such a Human is already a Human who has a User
with elevated permissions, as you have to be to assign Roles to Users
Projects.

In any case, as I mentioned in the other mail, I think there are a bunch
of details here that are going to require us being in the room - and
everyone realizing that everyones use cases and everyones concerns are
important. If we dig in, I'm sure we can come out on the other side with
happiness and joy.

I'm trying to avoid taking a side here because everyone is right.
++

Currently anybody who want to do anything remotely 'cloudy' (i.e. have
the application talk to OpenStack APIs) has to either share their
personal password with the application (and by extension their whole

Or - create an account in the Team's name and by storing the password
for that account realizing that everyone on the team has access to the
password so that a user-leaving-team event necessitates a "quick, let's
go re-key everything" event.

team) or to do the thing that is the polar opposite of cloud: file a
ticket with IT to get a service user account added and share that password instead.
Yup. Pretty much same thing.

And this really is a disaster for
OpenStack. On the other hand, allowing the creation of persistent
application credentials in the absence of regular automatic rotation
does create risk for those folks who are not aggressively auditing them
(perhaps because they have no legitimate use for them) and the result is
likely to be lots of clouds disabling them by policy, keeping their
users in the dark age of IT-ticket-filing <head...desk> and frustrating
our interoperability goals.

Yup.

It is possible in theory to satisfy both via the 'instance users'
concept, but the Nova team's response to this has consistently been
"prove to us that this has to be in Nova". Well, here's your answer.

Instance users do not solve this. Instance users can be built with this-
but instance users are themselves not sufficient. Instance users are
only sufficient in single-cloud ecosystems where it is possible to grant
permissions on all the resources in the single-cloud ecosystem to an
instance. We are not a single-cloud ecosystem.

Nodepool runs in Rackspace's DFW region. It has accounts across nine
different clouds. If this were only solved with Instance users we'd have
to boot a VM in each cloud so that we could call the publicly-accessible
REST APIs of the clouds to boot VMs in each cloud.

If I have to boot a VM so that I can boot a VM I swear to god I'm going
to rage-quit tech and raise goats.

Our current system that does not have a per-cloud VM-launching-VM
handles ~20k VM lifecycles per day load and it doesn't have to deal with
cross-WAN clustering concerns. Putting a password in a config file is
better than adding a fleet of VM-launching-VMs - we don't lose Infra
Root members THAT often.

Application Credentials, on the other hand, would allow us improve the
situation without imposing an architecture on us that is a copy of an
architecture from some clouds that have a vested interest in keeping
people thinking about problems in a uni-cloud manner.

Long-winded way of saying that I think if we get this right on the App
Creds side, we can actually solve all the use cases, including but not
limited to the Instance User use case.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 20, 2017 by Monty_Taylor (22,780 points)   2 4 7
0 votes

On 07/19/2017 10:00 PM, Adrian Turjak wrote:
The problem is then entirely procedural within a team. Do they rotate
all keys when one person leaves? Anything less is the same problem. All
we can do is make rotation less of a pain, but it will still be painful
no matter what, and depending on the situation the team makes the choice
of how to handle rotation if at all.

The sole reason for project level ownership of these application
credentials is so that a user leaving/being deleted isn't a scramble to
replace keys, and a team has the option/time to do it if they care about
the possibility of that person having known the keys (again, not our
problem, not a security flaw in code). Anything else, pretty much makes
this feature useless for teams. :(

Having both options (owned by project vs user) is useful, but the
'security issues' are kind of implied by using project owned app creds.
It's a very useful feature with some 'use at your own risk' attached.

I think this is a pretty good summary.

In many situations the situation of removing people from projects
(termination) will also physically remove their path to said clouds (as
they are beyond the firewall). It's not true with public clouds, but
it's not making the situation any worse, because right now it's shared
passwords to accounts.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 20, 2017 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 07/19/2017 09:27 PM, Monty Taylor wrote:
On 07/19/2017 12:18 AM, Zane Bitter wrote:

On 18/07/17 10:55, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?

If I'm understanding correctly, this would make application
credentials dependent on several cycles of policy work. Right?

I think having the ability to communicate deprecations though
oslo.policy would help here. We could use it to move towards better
default roles, which requires being able to set minimum privileges.

Using the current workflow requires operators to define the minimum
privileges for whatever is using the application credential, and
work that into their policy. Is that the intended workflow that we
want to put on the users and operators of application credentials?

The plan is to add an authorisation mechanism that is user-controlled
and independent of the (operator-controlled) policy. The beginnings
of this were included in earlier drafts of the spec, but were removed
in patch set 19 in favour of leaving them for a future spec:

https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst

Yes - that's right - and I expect to start work on that again as soon
as this next keystoneauth release with version discovery is out the door.

It turns out there are different POVs on this topic, and it's VERY
important to be clear which one we're talking about at any given point
in time. A bunch of the confusion just in getting as far as we've
gotten so far came from folks saying words like "policy" or "trusts"
or "ACLs" or "RBAC" - but not clarifying which group of cloud users
they were discussing and from what context.

The problem that Zane and I are are discussing and advocating for are
for UNPRIVILEDGED users who neither deploy nor operate the cloud but
who use the cloud to run applications.

Unfortunately, neither the current policy system nor trusts are useful
in any way shape or form for those humans. Policy and trusts are tools
for cloud operators to take a certain set of actions.

Similarly, the concern from the folks who are not in favor of
project-lifecycled application credentials is the one that Zane
outlined - that there will be $someone with access to those
credentials after a User change event, and thus $security will be
compromised.

There is a balance that can and must be found. The use case Zane and I
are talking about is ESSENTIAL, and literally ever single human who is
a actually using OpenStack to run applications needs it. Needed it
last year in fact, and they are, in fact doing things like writing
ssh-agent like daemons in which they can store their corporate LDAP
credentials so that their automation will work because we're not
giving them a workable option.

That said, the concerns about not letting a thing out the door that is
insecure by design like PHP4's globally scoped URL variables is also
super important.

So we need to find a design that meets both goals.

I have thoughts on the topic, but have been holding off until
version-discovery is out the door. My hunch is that, like application
credentials, we're not going to make significant headway without
getting humans in the room - because the topic is WAY too fraught with
peril.

I propose we set aside time at the PTG to dig in to this. Between Zane
and I and the Keystone core team I have confidence we can find a way out.

Done. I've added this thread to keystone's planning etherpad under
cross-project things we need to talk about [0]. Feel free to elaborate
and fill in context as you see fit. I'll make sure the content makes
it's way into a dedicated etherpad before we have that discussion
(usually as I go through each topic and plan the schedule).

[0] https://etherpad.openstack.org/p/keystone-queens-ptg

Monty

PS. It will not help to solve limited-scope before we solve this.
Limited scope is an end-user opt-in action and having it does not
remove the concerns that have been expressed.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 20, 2017 by Lance_Bragstad (11,080 points)   2 3 4
...