settingsLogin | Registersettings

Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

0 votes

fix subject typo

On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang zhang.lei.fly@gmail.com
wrote:

Kolla have support keystone fernet keys. But there are still some
topics worth to talk.

The key issue is key distribution. Kolla's solution is like

  • there is a task run frequently by cronjob to check whether
    the key should be rotate. This is controlled by
    fernet_token_expiry variable
  • When key rotate is required, the task in cron job will generate a
    new key by using keystone-manage fernet-rotate and distribute all
    keys in /etc/keystone/fernet-keys folder to other by using
    rsync --delete

one issue is: there is no global lock in rotate and distribute steps.
above command is ran on all controllers. it may cause issues if
all controllers run this at the same time.

Since we are using Ansible as deployment tools. there is not daemon
agent at all to keep rotate and distribution atomic. Is there any
easier way to implement a global lock?

possible solution:
1. configure cron job with different time on each controller
2. implement a global lock? ( no idea how )

[0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 6, 2017 in openstack-dev by Lei_Zhang (6,360 points)   1 2 5

4 Responses

0 votes

Two initial ideas:

We could create a specific ansible task to rotate the keys, and document
that operator should set up a cron job on the deployment node to run
this periodically.

We could also look at making use of VRRP (keepalived). Potentially the
cron job could run on every controller, but only take action if it
identifies it's the one with the VIP.

The second seems preferable to me as it requires no additional effort on
the part of the operator. Maybe there's problems with this though that
I'm not thinking of.

-Paul

On 06/03/17 05:52, Jeffrey Zhang wrote:
fix subject typo

On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang <zhang.lei.fly@gmail.com
zhang.lei.fly@gmail.com> wrote:

Kolla have support keystone fernet keys. But there are still some
topics worth to talk.

The key issue is key distribution. Kolla's solution is like

* there is a task run frequently by cronjob to check whether
  the key should be rotate. This is controlled by
  `fernet_token_expiry` variable
* When key rotate is required, the task in cron job will generate a
  new key by using `keystone-manage fernet-rotate` and distribute all
  keys in /etc/keystone/fernet-keys folder to other by using
  `rsync --delete`

one issue is: there is no global lock in rotate and distribute steps.
above command is ran on all controllers. it may cause issues if
all controllers run this at the same time.

Since we are using Ansible as deployment tools. there is not daemon
agent at all to keep rotate and distribution atomic. Is there any
easier way to implement a global lock?

possible solution:
1. configure cron job with different time on each controller
2. implement a global lock? ( no idea how )

[0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html
<https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html>

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me 

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 6, 2017 by Paul_Bourke (3,840 points)   1 3
0 votes

I don't think it would cause an issue if every controller rotated all at
once. The issues are more along the lines of rotating to key C when there
are tokens out there that are encrypted with keys A and B. In other words
over-rotation. As long as your keys are properly staged, do the rotation
all at once or space them out, should not make any difference.

On Sun, Mar 5, 2017 at 10:52 PM, Jeffrey Zhang zhang.lei.fly@gmail.com
wrote:

fix subject typo

On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang zhang.lei.fly@gmail.com
wrote:

Kolla have support keystone fernet keys. But there are still some
topics worth to talk.

The key issue is key distribution. Kolla's solution is like

  • there is a task run frequently by cronjob to check whether
    the key should be rotate. This is controlled by
    fernet_token_expiry variable
  • When key rotate is required, the task in cron job will generate a
    new key by using keystone-manage fernet-rotate and distribute all
    keys in /etc/keystone/fernet-keys folder to other by using
    rsync --delete

one issue is: there is no global lock in rotate and distribute steps.
above command is ran on all controllers. it may cause issues if
all controllers run this at the same time.

Since we are using Ansible as deployment tools. there is not daemon
agent at all to keep rotate and distribution atomic. Is there any
easier way to implement a global lock?

possible solution:
1. configure cron job with different time on each controller
2. implement a global lock? ( no idea how )

[0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 6, 2017 by Matt_Fischer (9,340 points)   1 4 7
0 votes

On Mon, Mar 6, 2017 at 6:05 PM, Paul Bourke paul.bourke@oracle.com wrote:

Two initial ideas:

We could create a specific ansible task to rotate the keys, and document
that operator should set up a cron job on the deployment node to run this
periodically.

We could also look at making use of VRRP (keepalived). Potentially the
cron job could run on every controller, but only take action if it
identifies it's the one with the VIP.

The second seems preferable to me as it requires no additional effort on
the part of the operator. Maybe there's problems with this though that I'm
not thinking of.

-Paul

​Thanks Paul, ​

​second seems better. We can implement a file lock to ensure only one
rotate and distribute process is running at the same time.

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 7, 2017 by Lei_Zhang (6,360 points)   1 2 5
0 votes

On Tue, Mar 7, 2017 at 2:09 AM, Matt Fischer matt@mattfischer.com wrote:

I don't think it would cause an issue if every controller rotated all at
once. The issues are more along the lines of rotating to key C when there
are tokens out there that are encrypted with keys A and B. In other words
over-rotation. As long as your keys are properly staged, do the rotation
all at once or space them out, should not make any difference.

​The issue is "at once".
It takes some time to rotate and distribute the keys. There is one case
that.
controller A and controller B generate a new different keys. Then they copy
the ​key to other by using rsync.

A: 0 1 2 3
B: 0' 1' 2 3

When distributing, the 0/0' and 1/1' may be overrode(rsync hold the delete
file handler and copy it to other one). it will lead to

A: 0' 1' 2 3
B: 0 1 2 3

next rotation, it may become

A: 0' 1' 2' 3
B: 0 1 2 3

after distribute , it become

A: 0 1 2 3
B: 0' 1' 2' 3

Next rotation and distribute, issue happen.

This is a small probability, but it still possible.

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 7, 2017 by Lei_Zhang (6,360 points)   1 2 5
...