settingsLogin | Registersettings

[openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

0 votes

hi,

we're trying to address in TripleO a couple of use cases for which we'd
like to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift
rings, which need some data related to the Heat stack (like the list of
Swift nodes and their disks), so it can't be executed in advance, yet it
provides data which is needed to complete successfully the deployment of
the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

I was wondering if it would make sense to have a property for the
existing Workflow resource to let the user decide if the workflow should
also be triggered on CREATE/UPDATE? And if it would make sense to
block the Workflow resource until the execution result is returned in
that case?

Alternatively, would an ex-novo Execution resource make more sense?

Or are there different ideas, approaches to the problem?

--
Giulio Fidente
GPG KEY: 08D733BA


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jan 20, 2017 in openstack-dev by Giulio_Fidente (3,980 points)   4 4

7 Responses

0 votes

we're trying to address in TripleO a couple of use cases for which we'd
like to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift
rings, which need some data related to the Heat stack (like the list of
Swift nodes and their disks), so it can't be executed in advance, yet it
provides data which is needed to complete successfully the deployment of
the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

I was wondering if it would make sense to have a property for the
existing Workflow resource to let the user decide if the workflow should
also be triggered on CREATE/UPDATE? And if it would make sense to
block the Workflow resource until the execution result is returned in
that case?

I think it needs to be triggered a bit later actually? For the Swift use
case it needs to be executed after all instances are created (but
preferably before starting any Puppet actions on the nodes), not when
the CREATE/UPDATE itself actually starts.

Alternatively, would an ex-novo Execution resource make more sense?

Or are there different ideas, approaches to the problem?

As a workaround for now I'm using the signal URL and trigger it in a
shell script on the nodes (the shell script is running anyways to fetch
and validate the rings). To avoid multiple parallel workflow executions
triggered by a dozen nodes I set a flag in the Mistral environment;
further actions will immediately return then.

I'd prefer a different and cleaner approach like you proposed but for me
that's working well for the moment.

-- Christian


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by cschwede_at_redhat.c (600 points)   2
0 votes

On 12/16/2016 01:56 PM, Christian Schwede wrote:

we're trying to address in TripleO a couple of use cases for which we'd
like to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift
rings, which need some data related to the Heat stack (like the list of
Swift nodes and their disks), so it can't be executed in advance, yet it
provides data which is needed to complete successfully the deployment of
the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

I was wondering if it would make sense to have a property for the
existing Workflow resource to let the user decide if the workflow should
also be triggered on CREATE/UPDATE? And if it would make sense to
block the Workflow resource until the execution result is returned in
that case?

I think it needs to be triggered a bit later actually? For the Swift use
case it needs to be executed after all instances are created (but
preferably before starting any Puppet actions on the nodes), not when
the CREATE/UPDATE itself actually starts.

yep, I was referring to the workflow resource CREATE/UPDATE action

we have complete control in Heat about when the workflow resource itself
should be created

Alternatively, would an ex-novo Execution resource make more sense?

Or are there different ideas, approaches to the problem?

As a workaround for now I'm using the signal URL and trigger it in a
shell script on the nodes (the shell script is running anyways to fetch
and validate the rings). To avoid multiple parallel workflow executions
triggered by a dozen nodes I set a flag in the Mistral environment;
further actions will immediately return then.

I'd prefer a different and cleaner approach like you proposed but for me
that's working well for the moment.

ack
--
Giulio Fidente
GPG KEY: 08D733BA


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by Giulio_Fidente (3,980 points)   4 4
0 votes

On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente gfidente@redhat.com wrote:
hi,

we're trying to address in TripleO a couple of use cases for which we'd like
to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift rings,
which need some data related to the Heat stack (like the list of Swift nodes
and their disks), so it can't be executed in advance, yet it provides data
which is needed to complete successfully the deployment of the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

You can trigger it out of band with resource signal. But you're right,
Heat won't wait for the result.

I was wondering if it would make sense to have a property for the existing
Workflow resource to let the user decide if the workflow should also be
triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

Or are there different ideas, approaches to the problem?

If you could define the event outside of Heat (in your example,
publish something when a swift node is available), then you could use
Zaqar to trigger your workflow. If you want Heat to block that won't
do it though.

--
Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by therve_at_redhat.com (2,620 points)   1 2
0 votes

On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:
On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente gfidente@redhat.com wrote:

hi,

we're trying to address in TripleO a couple of use cases for which we'd like
to trigger a Mistral workflow from a Heat template.

One example where this would be useful is the creation of the Swift rings,
which need some data related to the Heat stack (like the list of Swift nodes
and their disks), so it can't be executed in advance, yet it provides data
which is needed to complete successfully the deployment of the overcloud.

Currently we can create a workflow from Heat, but we can't trigger its
execution and also we can't block Heat on the result of the execution.

You can trigger it out of band with resource signal. But you're right,
Heat won't wait for the result.

I was wondering if it would make sense to have a property for the existing
Workflow resource to let the user decide if the workflow should also be
triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

Well, they're already conflated in the mistral workflow resource, because
we allow creating an execution by sending a signal:

https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

That satisfies the requirement for asynchronous (signal/alarm driven)
workflow execution, but we need the synchronous version that can be fully
integrated into the heat stack lifecycle (without external dependencies on
alarms etc).

I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest). However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handlesignal and add a boolean to each handlefoo function), I think this
discussion warrants revisiting.

Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

Yeah I think whichever approach we take, a list of actions similar to
SoftwareDeployment makes sense, then you can elect to run a specific
workflow at any state transition in the lifecycle.

Or are there different ideas, approaches to the problem?

If you could define the event outside of Heat (in your example,
publish something when a swift node is available), then you could use
Zaqar to trigger your workflow. If you want Heat to block that won't
do it though.

Yeah that doesn't solve our use-case, we want to run a workflow during an
overcloud stack create, wait for the result, then continue (or fail).

Thanks!

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by Steven_Hardy (16,900 points)   2 7 12
0 votes

On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy shardy@redhat.com wrote:
On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:

On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente gfidente@redhat.com wrote:

I was wondering if it would make sense to have a property for the existing
Workflow resource to let the user decide if the workflow should also be
triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

Well, they're already conflated in the mistral workflow resource, because
we allow creating an execution by sending a signal:

https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

Right, I mentioned that elsewhere. But it doesn't change the resource
interface, for better or worse.

That satisfies the requirement for asynchronous (signal/alarm driven)
workflow execution, but we need the synchronous version that can be fully
integrated into the heat stack lifecycle (without external dependencies on
alarms etc).

I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest). However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handlesignal and add a boolean to each handlefoo function), I think this
discussion warrants revisiting.

It's unclear what changed since the discussion happened, just that you
have a use case without another approach possible?

Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

Yeah I think whichever approach we take, a list of actions similar to
SoftwareDeployment makes sense, then you can elect to run a specific
workflow at any state transition in the lifecycle.

Among the various solutions, I think that's the one I like the best
for now. It doesn't touch the workflow resource interface, and it
seems to fit relatively naturally (an API to call, a state to check,
etc).

Or are there different ideas, approaches to the problem?

If you could define the event outside of Heat (in your example,
publish something when a swift node is available), then you could use
Zaqar to trigger your workflow. If you want Heat to block that won't
do it though.

Yeah that doesn't solve our use-case, we want to run a workflow during an
overcloud stack create, wait for the result, then continue (or fail).

Noted.

--
Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by therve_at_redhat.com (2,620 points)   1 2
0 votes

On Fri, Dec 16, 2016 at 04:02:51PM +0100, Thomas Herve wrote:
On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy shardy@redhat.com wrote:

On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:

On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente gfidente@redhat.com wrote:

I was wondering if it would make sense to have a property for the existing
Workflow resource to let the user decide if the workflow should also be
triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

Well, they're already conflated in the mistral workflow resource, because
we allow creating an execution by sending a signal:

https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

Right, I mentioned that elsewhere. But it doesn't change the resource
interface, for better or worse.

That satisfies the requirement for asynchronous (signal/alarm driven)
workflow execution, but we need the synchronous version that can be fully
integrated into the heat stack lifecycle (without external dependencies on
alarms etc).

I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest). However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handlesignal and add a boolean to each handlefoo function), I think this
discussion warrants revisiting.

It's unclear what changed since the discussion happened, just that you
have a use case without another approach possible?

Well, honestly yes - my perspective has changed somewhat as I've had recent
first-hand exposure to several problems which would be very nicely solved
by the additional Execution resource (or interface) under discussion.

Also, it's unclear if the hinted-at future model involving zaqar
notifications actually solves this (or at least not in it's current form),
so I'm looking at this from a pragmatic standpoint and trying to find a
relatively low-impact way to just solve the problem and move forward.

E.g in https://review.openstack.org/#/c/267770/ the execution resource was
rejected based partly on the expectation of a future Zaqar driven model
satisfying the same requirement, which was futher described in this summit
talk:

https://www.openstack.org/videos/video/building-self-healing-applications-with-aodh-zaqar-and-mistral

While the patterns discussed there are definitely compelling, they still
don't address the requirement of wanting to just run some mistral workflow
as part of the heat stack lifecycle?

AFAICT it's describing an independent (of Heat) system which can take
actions in an event/alarm driven manner e.g the interaction between
telemetry services, Zaqar and Mistral are configured by, but essentially
asynchronous to Heat - all great, but doesn't solve our problem?

Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

Yeah I think whichever approach we take, a list of actions similar to
SoftwareDeployment makes sense, then you can elect to run a specific
workflow at any state transition in the lifecycle.

Among the various solutions, I think that's the one I like the best
for now. It doesn't touch the workflow resource interface, and it
seems to fit relatively naturally (an API to call, a state to check,
etc).

Yeah, honestly I think it's a simple step which some folks will find
useful, and it's likely to be low-imact from a heat perspective (folks can
continue to prefer the Aodh/Zaqar approach if it suits their use-case
better).

As always, open to suggestions of a better way to approach this, but atm
I'm feeling like just going ahead with the WorkflowExecution resource
(ensuring it can handle all lifecycle actions, e.g just like
SoftwareDeployment/SoftwareCompoinent which is quite conceptually similar,
as noted in 267770 ...) is the most workable option.

Thanks!

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Dec 16, 2016 by Steven_Hardy (16,900 points)   2 7 12
0 votes

Better late than never...

On 16/12/16 08:57, Steven Hardy wrote:

I know we've previously tried to steer execute/run type actions to signal
driven interfaces (and I myself have opposed these kinds of resources in
the past, to be honest). However, I can't currently see a better way to
handle this requirement, and given it may be pretty easy to fix (refactor
handlesignal and add a boolean to each handlefoo function), I think this
discussion warrants revisiting.

I think we need to get away from thinking of this as "I just need to run
a script in the middle of Heat's workflow, let me do it!". That's verb
thinking and that's the reason I -1'd
https://review.openstack.org/267770 originally.

We need to think of it as "I have some external thing that I need to
manage, but I can't teach Heat about it because it means nothing to my
cloud operator". That's noun thinking and something we can actually work
with.

For that reason I started talking to a few folks a while back about
implementing an equivalent of CloudFormation's "CustomResource"[1].
Basically, it sends a message to an SNS topic (c.f. Zaqar notification)
and waits for a success/failure response. We have the technology now to
trigger Mistral workflows from Zaqar messages.[2] It's clear that this
is both conceptually sound and solves a number of problems with the best
available solution, which is triggering workflows via Zaqar
notifications from user hooks:
- There's no way to fail a particular resource.
- The Zaqar queue has to be passed in to the environment when creating
the stack. This means that the queue has to be created outside of the stack.
- All stacks in the tree share the same queue for hook notifications.

One reason that CloudFormation implements CustomResource as a message to
a topic is that it allows integration with third-party service
providers. That's a valid use case, but I think there's an equally or
more valid use case for integrating with stuff in your own tenant that
is just under the end user's control rather than the operators. For that
case, it makes sense to cut out the Zaqar middleman and just trigger the
Mistral execution directly.

So I am on board with a Mistral execution resource, BUT:

  • I don't think we should call it a Mistral Execution. I think we
    should give it a name that indicates it's managing some external thing
    by executing Mistral workflows.
  • We need to learn the lesson of SoftwareComponent and include all
    the different phases of the lifecycle in the same resource. The user
    should not have to assemble the different phases (Create/Update/Delete)
    of their external resource using discrete workflows, like they do with
    SoftwareConfigs. It's too much to ask them to understand how to build
    the dependency graphs correctly - it took me over a year to work it
    out[3] and that was my whole job.
  • We should also learn the lesson of
    https://launchpad.net/bugs/1595040 and implement a mechanism to allow
    the template author to select (in advance) between update-in-place and
    update-replace for any given update from the get-go.

I also left similar comments on https://review.openstack.org/#/c/420664/
but I wanted to go into the background in more depth here.

We may also want to implement something like CustomResource in the
future, but I agree that this is the higher priority.

cheers,
Zane.

[1]
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
[2]
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Zaqar::MistralTrigger
[3]
https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=8b732241df6007c3005b14413ee1fe047eb4d108


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 20, 2017 by Zane_Bitter (21,640 points)   4 6 9
...