settingsLogin | Registersettings

[openstack-dev] [TripleO][Heat] Overcloud software updates and ResourceGroups

0 votes

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an
impedance mismatch with Heat concepts and how Heat runs its workflow. As
many talented TripleO-ers who have gone before can probably testify,
that's surprisingly difficult to do, but we did come up with an idea
that I think might work and which I'd like to get wider feedback on. For
clarity, I'm speaking here in the context of the new
overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update
some software on the server. (The exact mechanism for the update is not
important for this discussion; suffice to say that in principle it could
be as simple as "[yum|apt-get] update".) The SoftwareConfig would have
at least one input, though it need not do anything with the value.

Then each server has that config deployed to it with a
SoftwareDeployment at the time it is created. However, it is set to
execute only on the UPDATE action. The value of (one of) the input(s) is
obtained from a parameter.

As a result, we can trigger the software update by simply changing the
value of the input parameter, and the regular Heat dependency graph will
be respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding
update, you could even automatically handle rollbacks in the event the
user decided to cancel the update.)

And now we should be able to trigger an update to all of our servers, in
the regular Heat dependency order, by simply (thanks to the fact that
parameters now keep their previous values on stack updates unless
they're explicitly changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can
always obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So,
for example, all of the controller nodes would be updated in parallel,
with the likely result that the overcloud could be unavailable for some
time even if it is deployed with HA.

The good news is that a solution to this problem is already implemented
in Heat: rolling updates. For example, the controller node availability
problem can be solved by setting a rolling update batch size of 1. The
bad news is that rolling updates are implemented only for
AutoscalingGroups, not ResourceGroups.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than
the change from merge.py over to overcloud-without-mergepy), but that
also means that there'll never be a better time than now to make it.

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we
carefully designed that to work with both ResourceGroup and
AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/).
Steve, I think this was only implemented for ResourceGroup? An
AutoscalingGroup version of this should be feasible though, or do we
have better ideas for how to solve it in that context?

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 2, 2015 in openstack-dev by Zane_Bitter (21,640 points)   4 6 9
retagged Apr 14, 2015 by admin

8 Responses

0 votes

I'm not sure how to feel about this... Its clever...

It kind of feels like your really trying to be able to register 'actions' in heat so that heat users can poke the vm's to do something... For example "perform a chef run".

While using stack updates listed below could be made to work, is that trying to fit a square peg in a round hole? Would it be better to add a separate api for that? Maybe in the end though, thats just is a matter of what command the user runs, rather then how it gets things done? It may be the same under the hood.

What about multiple update actions? Perhaps some types of updates could be run in parallel and others must be done serially? How would you let the Autoscaling group know which updates could run which way?

As for ResourceGroup vs AutoscalingGroup, It would be really good for ResourceGroup to support rolling updates properly too. Would it be very difficult to implement it there too?

While having the updates happen in the template dependency order is interesting, is that really the correct thing to do? Why not reverse order? I'm guessing it may totally depend on the software. Maybe some app needs the clients upgraded before the server, or the server upgraded before the clients? It may even be version specific? There may even be some steps that aren't obvious where to run them.... "update the clients, upgrade the server packages, stop the servers, run the db upgrade script on one of the servers, start up all the servers"

Maybe this is a good place to hook Mistral and Heat together. Heat would have an api that allows actions to be performed on vm's. It would not have any ordering. Mistral could then poke the heat actions api for the stack to assemble workflows... Or for tighter integration, maybe a CompoundAction resource is created that really is a Mistral workflow that pokes the action api, and the workflow was exposed right back through the Heat action api so users could invoke complicated workflows the same way as simple ones...

Thanks,
Kevin


From: Zane Bitter [zbitter@redhat.com]
Sent: Thursday, April 02, 2015 3:31 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [TripleO][Heat] Overcloud software updates and ResourceGroups

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an
impedance mismatch with Heat concepts and how Heat runs its workflow. As
many talented TripleO-ers who have gone before can probably testify,
that's surprisingly difficult to do, but we did come up with an idea
that I think might work and which I'd like to get wider feedback on. For
clarity, I'm speaking here in the context of the new
overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update
some software on the server. (The exact mechanism for the update is not
important for this discussion; suffice to say that in principle it could
be as simple as "[yum|apt-get] update".) The SoftwareConfig would have
at least one input, though it need not do anything with the value.

Then each server has that config deployed to it with a
SoftwareDeployment at the time it is created. However, it is set to
execute only on the UPDATE action. The value of (one of) the input(s) is
obtained from a parameter.

As a result, we can trigger the software update by simply changing the
value of the input parameter, and the regular Heat dependency graph will
be respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding
update, you could even automatically handle rollbacks in the event the
user decided to cancel the update.)

And now we should be able to trigger an update to all of our servers, in
the regular Heat dependency order, by simply (thanks to the fact that
parameters now keep their previous values on stack updates unless
they're explicitly changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can
always obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So,
for example, all of the controller nodes would be updated in parallel,
with the likely result that the overcloud could be unavailable for some
time even if it is deployed with HA.

The good news is that a solution to this problem is already implemented
in Heat: rolling updates. For example, the controller node availability
problem can be solved by setting a rolling update batch size of 1. The
bad news is that rolling updates are implemented only for
AutoscalingGroups, not ResourceGroups.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than
the change from merge.py over to overcloud-without-mergepy), but that
also means that there'll never be a better time than now to make it.

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we
carefully designed that to work with both ResourceGroup and
AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/).
Steve, I think this was only implemented for ResourceGroup? An
AutoscalingGroup version of this should be feasible though, or do we
have better ideas for how to solve it in that context?

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 3, 2015 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

hi there,

thanks for sharing this, I have a

On 04/03/2015 12:31 AM, Zane Bitter wrote:
A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud

[...]

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

[...]

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

leaving the ResourceGroup/AutoScalingGroup to more knowledgeable people
on a side and trying instead to translate the templating approach into
user features, if I read it correctly this would also make it possible to:

  1. perform a config update without a software update as long as the
    update_timestamp param remains unchanged

  2. perform software updates of each ResourceGroup independently from the
    others by using as many update_timestamp params

  3. use different update.sh scripts per ResourceGroup

are the above correct?

My single minor concern is about the update script itself which, if not
left for editing to the user but bundled instead with t-h-t , should be
clever enough to cope with different distros and distro versions because
we can't know that from the template ... but this can be achieved by
abstracting it on top of Puppet itself it seems (or whichever other
config management tool is deployed)
--
Giulio Fidente
GPG KEY: 08D733BA


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 3, 2015 by Giulio_Fidente (3,980 points)   4 4
0 votes

Hi,

On Fri, Apr 3, 2015 at 1:31 AM, Zane Bitter zbitter@redhat.com wrote:

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an impedance
mismatch with Heat concepts and how Heat runs its workflow. As many
talented TripleO-ers who have gone before can probably testify, that's
surprisingly difficult to do, but we did come up with an idea that I think
might work and which I'd like to get wider feedback on. For clarity, I'm
speaking here in the context of the new overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update
some software on the server. (The exact mechanism for the update is not
important for this discussion; suffice to say that in principle it could be
as simple as "[yum|apt-get] update".) The SoftwareConfig would have at
least one input, though it need not do anything with the value.

Then each server has that config deployed to it with a SoftwareDeployment
at the time it is created. However, it is set to execute only on the UPDATE
action. The value of (one of) the input(s) is obtained from a parameter.

As a result, we can trigger the software update by simply changing the
value of the input parameter, and the regular Heat dependency graph will be
respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding update,
you could even automatically handle rollbacks in the event the user decided
to cancel the update.)

And now we should be able to trigger an update to all of our servers, in
the regular Heat dependency order, by simply (thanks to the fact that
parameters now keep their previous values on stack updates unless they're
explicitly changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can always
obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So, for
example, all of the controller nodes would be updated in parallel, with the
likely result that the overcloud could be unavailable for some time even if
it is deployed with HA.

The good news is that a solution to this problem is already implemented in
Heat: rolling updates. For example, the controller node availability
problem can be solved by setting a rolling update batch size of 1. The bad
news is that rolling updates are implemented only for AutoscalingGroups,
not ResourceGroups.

It seems we should implement rollingupdates for ResourceGroup, and even
more, rolling
create too. Just a couple of days ago we had a chat with
Sahara's PTL, Sergey Lukjanov, and he was asking if there is a way to
create a number of resources in batches, but with a single call. Sahara
does not need autoscaling, so our idea was exactly that - rollingcreate
and rolling
update for ResourceGroups should solve this. Thus we have one
more use case for that, I'm going to raise a spec (or bug?) soon.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than
the change from merge.py over to overcloud-without-mergepy), but that also
means that there'll never be a better time than now to make it.

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we carefully
designed that to work with both ResourceGroup and AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/). Steve,
I think this was only implemented for ResourceGroup? An AutoscalingGroup
version of this should be feasible though, or do we have better ideas for
how to solve it in that context?

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Best regards,
Pavlo Shchelokovskyy
Software Engineer
Mirantis Inc
www.mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 3, 2015 by Pavlo_Shchelokovskyy (4,760 points)   5 5
0 votes

On 02/04/15 20:31, Fox, Kevin M wrote:
I'm not sure how to feel about this... Its clever...

That's... unfortunate ;)

Ideally this would sound like something that is a natural fit for Heat's
data model. The reason it's not is that config management tools like
Puppet are less sophisticated than Heat in terms of their modelling -
they only model the things that you want to change, and there's no way
to reverse a change.

That's not really a criticism of Puppet, it's kind of inevitable that
you can't represent the full complexity of a modern Linux system.
TripleO initially tried to get around this by doing image replacement
only, but it turns out that's too heavyweight to be practical. The real
long-term fix, at least for the application part of a deployment, is
most likely containers. However, we need to wait for the Kolla project
to become mature before we can deploy OpenStack using containers in TripleO.

I started to respond to the rest of your comments, but I decided not to
do so here, because this thread is about what we can/should do right
now, for TripleO, in Kilo, with Heat already in feature freeze.

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 6, 2015 by Zane_Bitter (21,640 points)   4 6 9
0 votes

On 03/04/15 01:29, Giulio Fidente wrote:
hi there,

thanks for sharing this, I have a

On 04/03/2015 12:31 AM, Zane Bitter wrote:

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud

[...]

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

[...]

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

leaving the ResourceGroup/AutoScalingGroup to more knowledgeable people
on a side and trying instead to translate the templating approach into
user features, if I read it correctly this would also make it possible to:

  1. perform a config update without a software update as long as the
    update_timestamp param remains unchanged

  2. perform software updates of each ResourceGroup independently from the
    others by using as many update_timestamp params

  3. use different update.sh scripts per ResourceGroup

are the above correct?

Yes, right on for all 3.

My single minor concern is about the update script itself which, if not
left for editing to the user but bundled instead with t-h-t , should be
clever enough to cope with different distros and distro versions because
we can't know that from the template ... but this can be achieved by
abstracting it on top of Puppet itself it seems (or whichever other
config management tool is deployed)

I picture it as a distro-specific thing that we might configure in the
Heat environment (maybe using getfile as in the example above).
Alternatively, I guess we could try to write a multi-platform script,
but I'm not sure _any
of the distros would actually thank us ;)

  • ZB


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 6, 2015 by Zane_Bitter (21,640 points)   4 6 9
0 votes

On Thu, Apr 02, 2015 at 06:31:39PM -0400, Zane Bitter wrote:
A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an impedance
mismatch with Heat concepts and how Heat runs its workflow. As many talented
TripleO-ers who have gone before can probably testify, that's surprisingly
difficult to do, but we did come up with an idea that I think might work and
which I'd like to get wider feedback on. For clarity, I'm speaking here in
the context of the new overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update some
software on the server. (The exact mechanism for the update is not important
for this discussion; suffice to say that in principle it could be as simple
as "[yum|apt-get] update".) The SoftwareConfig would have at least one
input, though it need not do anything with the value.

Then each server has that config deployed to it with a SoftwareDeployment at
the time it is created. However, it is set to execute only on the UPDATE
action. The value of (one of) the input(s) is obtained from a parameter.

As a result, we can trigger the software update by simply changing the value
of the input parameter, and the regular Heat dependency graph will be
respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding update,
you could even automatically handle rollbacks in the event the user decided
to cancel the update.)

And now we should be able to trigger an update to all of our servers, in the
regular Heat dependency order, by simply (thanks to the fact that parameters
now keep their previous values on stack updates unless they're explicitly
changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can always
obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So, for
example, all of the controller nodes would be updated in parallel, with the
likely result that the overcloud could be unavailable for some time even if
it is deployed with HA.

The good news is that a solution to this problem is already implemented in
Heat: rolling updates. For example, the controller node availability problem
can be solved by setting a rolling update batch size of 1. The bad news is
that rolling updates are implemented only for AutoscalingGroups, not
ResourceGroups.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than the
change from merge.py over to overcloud-without-mergepy), but that also means
that there'll never be a better time than now to make it.

I wonder if this is an opportunity to look at how we converge
AutoScalingGroup and ResourceGroup in Heat?

It seems like the main barrier to transparent (non destructive)
substitution of
AutoScalingGroup for ResourceGroup is the resource naming (e.g it's a short
UUID vs an index derived name) - could we add a property to
AutoScalingGroup which allowed optionally to use index based naming, such
that switching from ResourceGroup to ASG in a stack-update wouldn't replace
all the group members?

Another possible fudge if moving to ASG is impractical could be to use the
index in the script applying the update, such that an offset is introduced
between any updates which may cause service interruption (I know it's a
kludge, but e.g sleeping for a time derived from the group index before
doing the update would be an ugly-but-simple interim workaround for the
"all updated at once" problem you describe).

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we carefully
designed that to work with both ResourceGroup and AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/). Steve, I
think this was only implemented for ResourceGroup? An AutoscalingGroup
version of this should be feasible though, or do we have better ideas for
how to solve it in that context?

Yeah, I started looking at an alternative interface to achieve the same
thing, basically by enabling heat resource-signal to signal an
AutoScalingGroup directly (instead of only via a ScalingPolicy).

I need to revive this patch:

https://review.openstack.org/#/c/143496/

Then we'd need another patch adding support for a new signal payload which
specifies a specific resource for removal. This is related to the
"replace" interface I described in this thread, only a "remove" variant
which would enable a functional replacement fot the ResourceGroup
removalpolicies+resourcelist interface we currently provide.

http://lists.openstack.org/pipermail/openstack-dev/2014-December/053447.html

The other gap which occurs to me is that the AutoScalingGroup
implementation doesn't support the index_var feature of ResourceGroup, and
I know that some folks are expecting to use that in TripleO, e.g:

https://review.openstack.org/#/c/169937/

Again, it seems like figuring out a series of incremental steps to stop the
divergence of the two group resources is what we really need, such that in
future folks don't get pushed into an either/or choice which results in a
later breaking change.

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 7, 2015 by Steven_Hardy (16,900 points)   2 7 12
0 votes

On 07/04/15 05:13, Steven Hardy wrote:
On Thu, Apr 02, 2015 at 06:31:39PM -0400, Zane Bitter wrote:

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an impedance
mismatch with Heat concepts and how Heat runs its workflow. As many talented
TripleO-ers who have gone before can probably testify, that's surprisingly
difficult to do, but we did come up with an idea that I think might work and
which I'd like to get wider feedback on. For clarity, I'm speaking here in
the context of the new overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update some
software on the server. (The exact mechanism for the update is not important
for this discussion; suffice to say that in principle it could be as simple
as "[yum|apt-get] update".) The SoftwareConfig would have at least one
input, though it need not do anything with the value.

Then each server has that config deployed to it with a SoftwareDeployment at
the time it is created. However, it is set to execute only on the UPDATE
action. The value of (one of) the input(s) is obtained from a parameter.

As a result, we can trigger the software update by simply changing the value
of the input parameter, and the regular Heat dependency graph will be
respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding update,
you could even automatically handle rollbacks in the event the user decided
to cancel the update.)

And now we should be able to trigger an update to all of our servers, in the
regular Heat dependency order, by simply (thanks to the fact that parameters
now keep their previous values on stack updates unless they're explicitly
changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can always
obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So, for
example, all of the controller nodes would be updated in parallel, with the
likely result that the overcloud could be unavailable for some time even if
it is deployed with HA.

The good news is that a solution to this problem is already implemented in
Heat: rolling updates. For example, the controller node availability problem
can be solved by setting a rolling update batch size of 1. The bad news is
that rolling updates are implemented only for AutoscalingGroups, not
ResourceGroups.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than the
change from merge.py over to overcloud-without-mergepy), but that also means
that there'll never be a better time than now to make it.

I wonder if this is an opportunity to look at how we converge
AutoScalingGroup and ResourceGroup in Heat?

As long as it's not one of those insoluble opportunities.

It seems like the main barrier to transparent (non destructive)
substitution of
AutoScalingGroup for ResourceGroup is the resource naming (e.g it's a short
UUID vs an index derived name) - could we add a property to
AutoScalingGroup which allowed optionally to use index based naming, such
that switching from ResourceGroup to ASG in a stack-update wouldn't replace
all the group members?

I would say the main barrier is that you can't ever change a resource's
type without replacing it, and even the hacky workaround we have
(abandon/adopt) is not robust enough to actually use. Resource naming
doesn't even make the list - AutoscalingGroup doesn't care how its
members are named and always preserves existing names.

Another possible fudge if moving to ASG is impractical could be to use the
index in the script applying the update, such that an offset is introduced
between any updates which may cause service interruption (I know it's a
kludge, but e.g sleeping for a time derived from the group index before
doing the update would be an ugly-but-simple interim workaround for the
"all updated at once" problem you describe).

If it comes to that I'd infinitely prefer setting pre-update hooks on
all the resources and control it from an external workflow. We have to
implement that anyway for other things (e.g. a phased initial
deployment). It'd just be nicer if Heat could handle that part for us.

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we carefully
designed that to work with both ResourceGroup and AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/). Steve, I
think this was only implemented for ResourceGroup? An AutoscalingGroup
version of this should be feasible though, or do we have better ideas for
how to solve it in that context?

Yeah, I started looking at an alternative interface to achieve the same
thing, basically by enabling heat resource-signal to signal an
AutoScalingGroup directly (instead of only via a ScalingPolicy).

I need to revive this patch:

https://review.openstack.org/#/c/143496/

Then we'd need another patch adding support for a new signal payload which
specifies a specific resource for removal. This is related to the
"replace" interface I described in this thread, only a "remove" variant
which would enable a functional replacement fot the ResourceGroup
removalpolicies+resourcelist interface we currently provide.

http://lists.openstack.org/pipermail/openstack-dev/2014-December/053447.html

Cool, I like this approach of passing data in a signal to remove/replace
a member. Wearing my downstream hat for a second, if we can make it
completely backwards compatible then I'd go so far as to say that
carrying it as a patch in RDO for Kilo, while distasteful, is probably
less distasteful to me than continuing with ResourceGroup.

The other gap which occurs to me is that the AutoScalingGroup
implementation doesn't support the index_var feature of ResourceGroup, and
I know that some folks are expecting to use that in TripleO, e.g:

https://review.openstack.org/#/c/169937/

I don't have enough context around that patch to know how it relates to
the index_var feature, and I don't see that feature being used anywhere
in tripleo-heat-templates.

What I do know is that anything that relies on an incrementing integer
index assigned by Heat is likely to run into problems eventually,
because the elision feature will inevitably cause the list to become
increasingly sparse over time. So we should probably pick another approach.

Again, it seems like figuring out a series of incremental steps to stop the
divergence of the two group resources is what we really need, such that in
future folks don't get pushed into an either/or choice which results in a
later breaking change.

I agree that Heat needs to stop maintaining two different
implementations of the exact same idea and adding new features to one or
the other, such that there's never one resource type that does all of
the things you need to do. However, it's not clear to me that there's a
path to seamlessly migrate both types to the same implementation
encompassing all of the features without breaking existing users.

So are you saying that you are -1 on converting the
overcloud-without-mergepy templates to AutoscalingGroup?

I haven't yet heard anyone say that a pre-Kilo breaking change like this
to overcloud-without-mergepy.yaml would be a problem for them...

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 7, 2015 by Zane_Bitter (21,640 points)   4 6 9
0 votes

On Tue, Apr 07, 2015 at 07:12:42PM -0400, Zane Bitter wrote:
On 07/04/15 05:13, Steven Hardy wrote:

On Thu, Apr 02, 2015 at 06:31:39PM -0400, Zane Bitter wrote:

A few of us have been looking for a way to perform software updates to
servers in a TripleO Heat/Puppet-based overcloud that avoids an impedance
mismatch with Heat concepts and how Heat runs its workflow. As many talented
TripleO-ers who have gone before can probably testify, that's surprisingly
difficult to do, but we did come up with an idea that I think might work and
which I'd like to get wider feedback on. For clarity, I'm speaking here in
the context of the new overcloud-without-mergepy templates.

The idea is that we create a SoftwareConfig that, when run, can update some
software on the server. (The exact mechanism for the update is not important
for this discussion; suffice to say that in principle it could be as simple
as "[yum|apt-get] update".) The SoftwareConfig would have at least one
input, though it need not do anything with the value.

Then each server has that config deployed to it with a SoftwareDeployment at
the time it is created. However, it is set to execute only on the UPDATE
action. The value of (one of) the input(s) is obtained from a parameter.

As a result, we can trigger the software update by simply changing the value
of the input parameter, and the regular Heat dependency graph will be
respected. The actual input value could be by convention a uuid, a
timestamp, a random string, or just about anything so long as it changes.

Here's a trivial example of what this deployment might look like:

updateconfig:
type: OS::Heat::SoftwareConfig
properties:
config: {get
file: doswupdate.sh}
inputs:
- name: updateaftertime
description: Timestamp of the most recent update request

updatedeployment:
type: OS::Heat::SoftwareDeployment
properties:
actions:
- UPDATE
config: {get
resource: updateconfig}
server: {get
resource: myserver}
input
values:
updateaftertime: {getparam: updatetimestamp}

(A possible future enhancement is that if you keep a mapping between
previous input values and the system state after the corresponding update,
you could even automatically handle rollbacks in the event the user decided
to cancel the update.)

And now we should be able to trigger an update to all of our servers, in the
regular Heat dependency order, by simply (thanks to the fact that parameters
now keep their previous values on stack updates unless they're explicitly
changed) running a command like:

heat stack-update myovercloud -f $TMPL -P "updatetimestamp=$(date)"

(A future goal of Heat is to make specifying the template again optional
too... I don't think that change landed yet, but in this case we can always
obtain the template from Tuskar, so it's not so bad.)

Astute readers may have noticed that this does not actually solve our
problem. In reality groups of similar servers are deployed within
ResourceGroups and there are no dependencies between the members. So, for
example, all of the controller nodes would be updated in parallel, with the
likely result that the overcloud could be unavailable for some time even if
it is deployed with HA.

The good news is that a solution to this problem is already implemented in
Heat: rolling updates. For example, the controller node availability problem
can be solved by setting a rolling update batch size of 1. The bad news is
that rolling updates are implemented only for AutoscalingGroups, not
ResourceGroups.

Accordingly, I propose that we switch the implementation of
overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This
would be a breaking change for overcloud updates (although no worse than the
change from merge.py over to overcloud-without-mergepy), but that also means
that there'll never be a better time than now to make it.

I wonder if this is an opportunity to look at how we converge
AutoScalingGroup and ResourceGroup in Heat?

As long as it's not one of those insoluble opportunities.

No, of course - all I'm suggesting is now might not be a bad time to
enumerate the areas of divergence, and figure out a medium term plan
towards, uh, convergence ;)

It seems like the main barrier to transparent (non destructive)
substitution of
AutoScalingGroup for ResourceGroup is the resource naming (e.g it's a short
UUID vs an index derived name) - could we add a property to
AutoScalingGroup which allowed optionally to use index based naming, such
that switching from ResourceGroup to ASG in a stack-update wouldn't replace
all the group members?

I would say the main barrier is that you can't ever change a resource's type
without replacing it, and even the hacky workaround we have (abandon/adopt)
is not robust enough to actually use. Resource naming doesn't even make the
list - AutoscalingGroup doesn't care how its members are named and always
preserves existing names.

Yeah, OK I've oversimplified - I was assuming we'd work out how to avoid
the type comparison, such that if you update any nested-stack based
resource we'd recurse and attempt to update the nested stack in-place, e.g
like we already do for all user-defined nested stacks.

Then, it does become a question of how can we (at least optionally) make
the templates generated by the two resources look the same, such that any
update becomes non-destructive.

Another possible fudge if moving to ASG is impractical could be to use the
index in the script applying the update, such that an offset is introduced
between any updates which may cause service interruption (I know it's a
kludge, but e.g sleeping for a time derived from the group index before
doing the update would be an ugly-but-simple interim workaround for the
"all updated at once" problem you describe).

If it comes to that I'd infinitely prefer setting pre-update hooks on all
the resources and control it from an external workflow. We have to implement
that anyway for other things (e.g. a phased initial deployment). It'd just
be nicer if Heat could handle that part for us.

Ok, agreed, I was just throwing some ideas out there, that one is a pretty
ugly suggestion ;)

I suspect that some folks (Tomas?) have possibly looked into this in the
past... can anybody identify any potential obstacles to the change? Two
candidates come to mind:

1) The SoftwareDeployments (plural) resource type. I believe we carefully
designed that to work with both ResourceGroup and AutoscalingGroup though.
2) The elision feature (https://review.openstack.org/#/c/128365/). Steve, I
think this was only implemented for ResourceGroup? An AutoscalingGroup
version of this should be feasible though, or do we have better ideas for
how to solve it in that context?

Yeah, I started looking at an alternative interface to achieve the same
thing, basically by enabling heat resource-signal to signal an
AutoScalingGroup directly (instead of only via a ScalingPolicy).

I need to revive this patch:

https://review.openstack.org/#/c/143496/

Then we'd need another patch adding support for a new signal payload which
specifies a specific resource for removal. This is related to the
"replace" interface I described in this thread, only a "remove" variant
which would enable a functional replacement fot the ResourceGroup
removalpolicies+resourcelist interface we currently provide.

http://lists.openstack.org/pipermail/openstack-dev/2014-December/053447.html

Cool, I like this approach of passing data in a signal to remove/replace a
member. Wearing my downstream hat for a second, if we can make it completely
backwards compatible then I'd go so far as to say that carrying it as a
patch in RDO for Kilo, while distasteful, is probably less distasteful to me
than continuing with ResourceGroup.

Ok, sounds good - I've updated https://review.openstack.org/#/c/143496 so
hopefully we can work towards landing it.

A subsequent patch defining some sort of "remove" signal payload should be
fairly easy to implement I think.

The other gap which occurs to me is that the AutoScalingGroup
implementation doesn't support the index_var feature of ResourceGroup, and
I know that some folks are expecting to use that in TripleO, e.g:

https://review.openstack.org/#/c/169937/

I don't have enough context around that patch to know how it relates to the
index_var feature, and I don't see that feature being used anywhere in
tripleo-heat-templates.

Yeah, to clarify, it's not used right now, but it seems likely we will want
to use it in the near future.

The patch Dan Sneddon has posted enables a node index to be passed into
os-net-config, such that it may be used to generate a unique IP address for
a given subnet definition - IMO we need to ensure any move to ASG doesn't
block that work.

What I do know is that anything that relies on an incrementing integer index
assigned by Heat is likely to run into problems eventually, because the
elision feature will inevitably cause the list to become increasingly sparse
over time. So we should probably pick another approach.

The main requirement is for a unique index, not necessarily a continuous
one, so provided the index is consistent (immutable for the lifetime of the
group member, as is currently the case with ResourceGroup) then it should
work OK AFAICT. Open to other ideas if you have them :)

Again, it seems like figuring out a series of incremental steps to stop the
divergence of the two group resources is what we really need, such that in
future folks don't get pushed into an either/or choice which results in a
later breaking change.

I agree that Heat needs to stop maintaining two different implementations of
the exact same idea and adding new features to one or the other, such that
there's never one resource type that does all of the things you need to do.
However, it's not clear to me that there's a path to seamlessly migrate both
types to the same implementation encompassing all of the features without
breaking existing users.

Ok, well I'm not suggesting we solve all those problems now, just that it'd
be helpful to enumerate them, and try to figure out if there is some path
we can take which avoids user-breakage.

So are you saying that you are -1 on converting the
overcloud-without-mergepy templates to AutoscalingGroup?

I haven't yet heard anyone say that a pre-Kilo breaking change like this to
overcloud-without-mergepy.yaml would be a problem for them...

No, I'm +1 on it, provided we can work out both the "elide" and index_var
problems.

Cheers,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 8, 2015 by Steven_Hardy (16,900 points)   2 7 12
...