settingsLogin | Registersettings

[openstack-dev] [TripleO] Driving workflows with Mistral

0 votes

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jan 11, 2016 in openstack-dev by Dan_Prince (8,160 points)   1 5 8

11 Responses

0 votes

----- Original Message -----
Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.

Hiya! Thanks for putting down your thoughts.

I think I fundamentally disagree with the idea of using Mistral, simply
because many of the actions we'd like to expose through a REST API
(described in the tripleo-common deployment library spec [1]) aren't
workflows; they're straightforward get/set methods. Putting a workflow
engine in front of that feels like overkill and an added complication
that simply isn't needed. And added complications can lead to unneeded
complications: for instance, one open Mistral bug details how it may
not scale well [2].

The Mistral solution feels like we're trying to force a circular peg
into a round-ish hole. In a vacuum, if we were to consider the
engineering problem of exposing a code base to outside consumers in a
non-language specific fashion - I'm pretty sure we'd just suggest the
creation of a REST API and be done with it; the thought of using a
workflow engine as the frontend would not cross our minds.

I don't really agree with the 'purist' argument. We already have custom
business logic written in the TripleO CLI; accepting that within TripleO,
but not a very thin API layer, feels like an arbitrary line to me. And
if that line exists, I'd argue that if it forces overcomplicated
solutions to straightforward engineering problems, then that line needs
to be moved.

Mainn

[1] https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst
[2] https://bugs.launchpad.net/mistral/+bug/1423054


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 11, 2016 by Tzu-Mainn_Chen (1,420 points)   4
0 votes

On 01/11/2016 11:09 PM, Tzu-Mainn Chen wrote:
----- Original Message -----

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.

Hiya! Thanks for putting down your thoughts.

I think I fundamentally disagree with the idea of using Mistral, simply
because many of the actions we'd like to expose through a REST API
(described in the tripleo-common deployment library spec [1]) aren't
workflows; they're straightforward get/set methods.

Right, because this spec describes nearly nothing from what is present
in tripleoclient now. And what we realistically have now is workflows,
which we'll have to reimplement in API somehow. So maybe we need both:
the get/set TripleO API for deployment plans and Mistral for workflows.

Putting a workflow
engine in front of that feels like overkill and an added complication
that simply isn't needed. And added complications can lead to unneeded
complications: for instance, one open Mistral bug details how it may
not scale well [2].

Let's not talk about scaling in the context of what we have in
tripleoclient now ;)

The Mistral solution feels like we're trying to force a circular peg
into a round-ish hole. In a vacuum, if we were to consider the
engineering problem of exposing a code base to outside consumers in a
non-language specific fashion - I'm pretty sure we'd just suggest the
creation of a REST API and be done with it; the thought of using a
workflow engine as the frontend would not cross our minds.

I don't really agree with the 'purist' argument. We already have custom
business logic written in the TripleO CLI; accepting that within TripleO,
but not a very thin API layer, feels like an arbitrary line to me. And
if that line exists, I'd argue that if it forces overcomplicated
solutions to straightforward engineering problems, then that line needs
to be moved.

Mainn

[1] https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst
[2] https://bugs.launchpad.net/mistral/+bug/1423054


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Dmitry_Tantsur (18,080 points)   2 4 8
0 votes

On 01/11/2016 04:51 PM, Dan Prince wrote:
Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g. access the
templates installed at undercloud's
/usr/share/openstack-tripleo-heat-templates?

I Mistral action able to call either OpenStack service python client or
OpenStack service API directly?

What is the response from the Mistral action in the workflow? Lets say
we'd use Mistral to get a list of available environments (we do this in
tripleo-common now) So we call Mistral API to trigger a workflow that
has single action which gets the list of environments. Is mistral able
to provide this list as a response, or it is able just to trigger a
workflow?

In similar manner, is Mistral able to provide a way to get workflow in
progress data? E.g. We have a Mistral workflow for nodes introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the workflow and
now it needs to have a way to track the progress of that workflow. How
would this be achieved? I think forcing GUI to poll the APIs that are
called in the actions and try to implement logic that estimates what
state the workflow is to report about it is not a valid solution. We
need the workflow API (Mistral) to provide a lets say web sockets
connection and push the status of actions along with relevant data, so
GUI can listen to those.

I am about to implement your nodes workflow in the GUI to test how it works.

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Jiri_Tomasek (2,240 points)   2 3
0 votes

On Tue, 2016-01-12 at 12:52 +0100, Jiri Tomasek wrote:
On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in
that
they have to duplicate deployment logic in potentially multiple
places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service
called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and
executing
them via an OpenStack API. Initially the effort was focused around
the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this
effort
[2]. This particular workflow seemed to fit nicely within the
Mistral
tooling.


More recently I've turned my attention to what it might look like
if we
were to use Mistral as a replacement for the TripleO API entirely.
This
brings forth the question of would TripleO be better off building
out
its own API... or would relying on existing OpenStack APIs be a
better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack
    API's
    we require [3]. Integration with keystone is baked in. Adding
    support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some
    of
    our more complex workflows (perhaps those that aren't easy to
    replicate
    with pure YAML workflows) by creating our own TripleO Mistral
    actions.
    This approach would be similar to creating a custom Heat
    resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again
    build
    out our YAML workflows to drive things. This might allow us to
    build
    off some of the tripleo-common consolidation that is already
    underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows
    get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely
    with
    the deploy OpenStack with OpenStack. I sort of feel like if we have
    to
    build our own API in TripleO part of this vision has failed and
    could
    even be seen as a massive technical debt which would likely be hard
    to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as
    new
    Mistral actions as well. I'm not convinced we require TripleO API
    just
    to support a validations mechanism yet. Perhaps validations seem
    hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity
    at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO
    that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to
enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo
from
Delorean so that you have the recently added Mistral packages for
this
to work. Although the feature is disable by default this should
enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/ac
tion
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g. access
the 
templates installed at undercloud's 
/usr/share/openstack-tripleo-heat-templates?

Yes. I'm actually working some prototype Mistral actions for tripleo-
common which will do just this to dovetail into a deployment workflow.
(I had to take care of a few heatclient things first though).

I Mistral action able to call either OpenStack service python client
or 
OpenStack service API directly?

The Mistral actions themselves would be written in Python. So I think
the answer here is yes... we could easily make use of existing
OpenStack python clients, or go at the API's ourselves.

What is the response from the Mistral action in the workflow? Lets
say 
we'd use Mistral to get a list of available environments (we do this
in 
tripleo-common now) So we call Mistral API to trigger a workflow
that 
has single action which gets the list of environments. Is mistral
able 
to provide this list as a response, or it is able just to trigger a 
workflow?

Yes, the response from individual actions is quite flexible and can
really be just about anything we want so long at it is serializable I
think.

Probably best to look directly at the Mistral action base class for the
best answer here I think though:

http://git.openstack.org/cgit/openstack/mistral/tree/mistral/actions/ba
se.py#n50

The action result could then be processed by YAQL in a Mistral workflow
and then used in subsequent workflow items or treated as an output to
the workflow itself.

In similar manner, is Mistral able to provide a way to get workflow
in 
progress data? E.g. We have a Mistral workflow for nodes
introspection. 
This workflow contains several actions calling to ironic and 
ironic-inspector apis. GUI calls Mistral API to trigger the workflow
and 
now it needs to have a way to track the progress of that workflow.
How 
would this be achieved?

I've been running 'mistral execution-list' and then you can watch
(poll) for the relevant execution items to finish running. Sure,
polling isn't great but I'd say lets start with this perhaps.

I think forcing GUI to poll the APIs that are 
called in the actions and try to implement logic that estimates what 
state the workflow is to report about it is not a valid solution. We 
need the workflow API (Mistral) to provide a lets say web sockets 
connection and push the status of actions along with relevant data,
so 
GUI can listen to those.

I don't think Mistral has a websockets implementation. I think Zaqar
does though, and I think perhaps one way we could go about this sort of
notification might be to integrate our workflows with a Zaqar queue or
something. GUI would listen to a Zaqar queue for example... and the
workflow (as it executes) would post things to a specific queue.
Perhaps this is opt-in, only if a Zaqar queue is provided to a given
workflow. FWIW, integrating Mistral w/ Zaqar actions would likely be
quite easy.

Alternately we could look at what it would take to add websocket
support to Mistral directly.

I am about to implement your nodes workflow in the GUI to test how it
works.

Jirka



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubs
cribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Dan_Prince (8,160 points)   1 5 8
0 votes

On 01/12/2016 06:52 AM, Jiri Tomasek wrote:
On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g. access the
templates installed at undercloud's
/usr/share/openstack-tripleo-heat-templates?

I believe with mistral there would be an intermediate step of uploading
the templates to Swift first. Heat can read templates from swift, and
any mistral workflows would be able to read the templates out, modify
them, and save back to swift.

Object stores FTW

I Mistral action able to call either OpenStack service python client or
OpenStack service API directly?

What is the response from the Mistral action in the workflow? Lets say
we'd use Mistral to get a list of available environments (we do this in
tripleo-common now) So we call Mistral API to trigger a workflow that
has single action which gets the list of environments. Is mistral able
to provide this list as a response, or it is able just to trigger a
workflow?

In similar manner, is Mistral able to provide a way to get workflow in
progress data? E.g. We have a Mistral workflow for nodes introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the workflow and
now it needs to have a way to track the progress of that workflow. How
would this be achieved? I think forcing GUI to poll the APIs that are
called in the actions and try to implement logic that estimates what
state the workflow is to report about it is not a valid solution. We
need the workflow API (Mistral) to provide a lets say web sockets
connection and push the status of actions along with relevant data, so
GUI can listen to those.

I can see a few options for progress polling/push.

Mistral:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Poll mistral to see how many are done

TripleO API
1. Build the REST API (we build)
1. Send requests to start introspection (we build)
1. Poll TripleO API until things are done

Mistral + Zaqar:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Send updates to Zaqar (we didn't have to build)
1. Get websocket updates to the GUI from Zaqar (we didn't have to build)

Zaqar is (of course) designed to deliver updates like this, so every
project on the face of the planet doesn't have to rebuild websocket
notifications, which is a good thing.

I am about to implement your nodes workflow in the GUI to test how it
works.

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Ryan_Brown (3,520 points)   2 2
0 votes

On 01/12/2016 08:24 AM, Dan Prince wrote:
On Tue, 2016-01-12 at 12:52 +0100, Jiri Tomasek wrote:

On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:
[snip]

In similar manner, is Mistral able to provide a way to get workflow
in
progress data? E.g. We have a Mistral workflow for nodes
introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the workflow
and
now it needs to have a way to track the progress of that workflow.
How
would this be achieved?

I've been running 'mistral execution-list' and then you can watch
(poll) for the relevant execution items to finish running. Sure,
polling isn't great but I'd say lets start with this perhaps.

I think forcing GUI to poll the APIs that are
called in the actions and try to implement logic that estimates what
state the workflow is to report about it is not a valid solution. We
need the workflow API (Mistral) to provide a lets say web sockets
connection and push the status of actions along with relevant data,
so
GUI can listen to those.

I don't think Mistral has a websockets implementation. I think Zaqar
does though, and I think perhaps one way we could go about this sort of
notification might be to integrate our workflows with a Zaqar queue or
something. GUI would listen to a Zaqar queue for example... and the
workflow (as it executes) would post things to a specific queue.
Perhaps this is opt-in, only if a Zaqar queue is provided to a given
workflow. FWIW, integrating Mistral w/ Zaqar actions would likely be
quite easy.

Heat currently has a similar system for stack actions. The user provides
a zaqar queue in the environment file, and every resource action
generates a notification on that queue. I think a similar system would
work for Mistral jobs.

For "domain-specific" notifications like receiving a notification for
certain discovery stages, the TripleO custom actions could also publish
messages to a queue so users get something more granular than "this
Mistral job finished."

Alternately we could look at what it would take to add websocket
support to Mistral directly.

I really don't think adding websockets to Mistral would be worth it -
Zaqar is a much better solution for notifications since it already has
websocket, webhook, and even email (yes, that's right) notification options.

I am about to implement your nodes workflow in the GUI to test how it
works.

Jirka

--
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Ryan_Brown (3,520 points)   2 2
0 votes

On 01/12/2016 04:22 PM, Ryan Brown wrote:
On 01/12/2016 06:52 AM, Jiri Tomasek wrote:

On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g. access the
templates installed at undercloud's
/usr/share/openstack-tripleo-heat-templates?

I believe with mistral there would be an intermediate step of
uploading the templates to Swift first. Heat can read templates from
swift, and any mistral workflows would be able to read the templates
out, modify them, and save back to swift.

Correct, but from the Mistral usage standpoint, having the flexibility
is a good thing IMO regardless of the example I have chosen.

Object stores FTW

I Mistral action able to call either OpenStack service python client or
OpenStack service API directly?

What is the response from the Mistral action in the workflow? Lets say
we'd use Mistral to get a list of available environments (we do this in
tripleo-common now) So we call Mistral API to trigger a workflow that
has single action which gets the list of environments. Is mistral able
to provide this list as a response, or it is able just to trigger a
workflow?

In similar manner, is Mistral able to provide a way to get workflow in
progress data? E.g. We have a Mistral workflow for nodes introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the workflow and
now it needs to have a way to track the progress of that workflow. How
would this be achieved? I think forcing GUI to poll the APIs that are
called in the actions and try to implement logic that estimates what
state the workflow is to report about it is not a valid solution. We
need the workflow API (Mistral) to provide a lets say web sockets
connection and push the status of actions along with relevant data, so
GUI can listen to those.

I can see a few options for progress polling/push.

Mistral:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Poll mistral to see how many are done

TripleO API
1. Build the REST API (we build)
1. Send requests to start introspection (we build)
1. Poll TripleO API until things are done

Mistral + Zaqar:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Send updates to Zaqar (we didn't have to build)
1. Get websocket updates to the GUI from Zaqar (we didn't have to build)

Zaqar is (of course) designed to deliver updates like this, so every
project on the face of the planet doesn't have to rebuild websocket
notifications, which is a good thing.

Thanks, this sounds very good. So the Mistral + Zaqar workflow means
that as part of workflow action, Mistral notifies Zaqar that certain
thing happened. Zaqar then notifies listening GUI providing information
to identify what API call the GUI needs to make to retrieve relevant
data or even provide the data itself (e.g. Mistral fails, some error
occurs etc.)

I am about to implement your nodes workflow in the GUI to test how it
works.

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Jiri_Tomasek (2,240 points)   2 3
0 votes

On 01/12/2016 10:50 AM, Jiri Tomasek wrote:
On 01/12/2016 04:22 PM, Ryan Brown wrote:

On 01/12/2016 06:52 AM, Jiri Tomasek wrote:

On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g. access the
templates installed at undercloud's
/usr/share/openstack-tripleo-heat-templates?

I believe with mistral there would be an intermediate step of
uploading the templates to Swift first. Heat can read templates from
swift, and any mistral workflows would be able to read the templates
out, modify them, and save back to swift.

Correct, but from the Mistral usage standpoint, having the flexibility
is a good thing IMO regardless of the example I have chosen.

Object stores FTW

I Mistral action able to call either OpenStack service python client or
OpenStack service API directly?

What is the response from the Mistral action in the workflow? Lets say
we'd use Mistral to get a list of available environments (we do this in
tripleo-common now) So we call Mistral API to trigger a workflow that
has single action which gets the list of environments. Is mistral able
to provide this list as a response, or it is able just to trigger a
workflow?

In similar manner, is Mistral able to provide a way to get workflow in
progress data? E.g. We have a Mistral workflow for nodes introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the workflow and
now it needs to have a way to track the progress of that workflow. How
would this be achieved? I think forcing GUI to poll the APIs that are
called in the actions and try to implement logic that estimates what
state the workflow is to report about it is not a valid solution. We
need the workflow API (Mistral) to provide a lets say web sockets
connection and push the status of actions along with relevant data, so
GUI can listen to those.

I can see a few options for progress polling/push.

Mistral:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Poll mistral to see how many are done

TripleO API
1. Build the REST API (we build)
1. Send requests to start introspection (we build)
1. Poll TripleO API until things are done

Mistral + Zaqar:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Send updates to Zaqar (we didn't have to build)
1. Get websocket updates to the GUI from Zaqar (we didn't have to build)

Zaqar is (of course) designed to deliver updates like this, so every
project on the face of the planet doesn't have to rebuild websocket
notifications, which is a good thing.

Thanks, this sounds very good. So the Mistral + Zaqar workflow means
that as part of workflow action, Mistral notifies Zaqar that certain
thing happened. Zaqar then notifies listening GUI providing information
to identify what API call the GUI needs to make to retrieve relevant
data or even provide the data itself (e.g. Mistral fails, some error
occurs etc.)

I don't know what the mistral-zaqar integration is at the moment, but if
it's not there you could post to zaqar in the custom TripleO actions.

I am about to implement your nodes workflow in the GUI to test how it
works.

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Jirka


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Ryan_Brown (3,520 points)   2 2
0 votes

On Tue, 2016-01-12 at 13:24 -0500, Ryan Brown wrote:
On 01/12/2016 10:50 AM, Jiri Tomasek wrote:

On 01/12/2016 04:22 PM, Ryan Brown wrote:

On 01/12/2016 06:52 AM, Jiri Tomasek wrote:

On 01/11/2016 04:51 PM, Dan Prince wrote:

Background info:

We've got a problem in TripleO at the moment where many of
our
workflows can be driven by the command line only. This causes
some
problems for those trying to build a UI around the workflows
in that
they have to duplicate deployment logic in potentially
multiple places.
There are specs up for review which outline how we might
solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack
service called
Mistral which contains a generic workflow API. Mistral
supports
defining workflows in YAML and then creating, managing, and
executing
them via an OpenStack API. Initially the effort was focused
around the
idea of creating a workflow in Mistral which could supplant
our
"baremetal introspection" workflow which currently lives in
python-
tripleoclient. I create a video presentation which outlines
this effort
[2]. This particular workflow seemed to fit nicely within the
Mistral
tooling.


More recently I've turned my attention to what it might look
like if we
were to use Mistral as a replacement for the TripleO API
entirely. This
brings forth the question of would TripleO be better off
building out
its own API... or would relying on existing OpenStack APIs be
a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the
    OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding
    support
    for new clients seems straightforward (I've had no issues in
    adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap
    some of
    our more complex workflows (perhaps those that aren't easy to
    replicate
    with pure YAML workflows) by creating our own TripleO Mistral
    actions.
    This approach would be similar to creating a custom Heat
    resource...
    something we have avoided with Heat in TripleO but I think it
    is
    perhaps more reasonable with Mistral and would allow us to
    again build
    out our YAML workflows to drive things. This might allow us
    to build
    off some of the tripleo-common consolidation that is already
    underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps
    workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits
    nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if
    we have to
    build our own API in TripleO part of this vision has failed
    and could
    even be seen as a massive technical debt which would likely
    be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be
    implemented as new
    Mistral actions as well. I'm not convinced we require TripleO
    API just
    to support a validations mechanism yet. Perhaps validations
    seem hard
    because we are simply trying to do them in the wrong places
    anyway?
    (like for example perhaps we should validate network
    connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral
    API more
    easy to interact with and expand upon. Perhaps this ends up
    being
    something that gets submitted as a patchset back to the
    TripleO that we
    accept into our upstream "stock" workflow sets.


Last week we landed the last patches [4] to our undercloud to
enable
installing Mistral by simply setting: enable_mistral = true
in
undercloud.conf. NOTE: you'll need to be using a recent trunk
repo from
Delorean so that you have the recently added Mistral packages
for this
to work. Although the feature is disable by default this
should enable
those wishing to tinker with Mistral as a new TripleO
undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mist
ral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workf
low



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-dev

Hi, I have a few questions:

Is Mistral action able to access/manipulate local files? E.g.
access the
templates installed at undercloud's
/usr/share/openstack-tripleo-heat-templates?

I believe with mistral there would be an intermediate step of
uploading the templates to Swift first. Heat can read templates
from
swift, and any mistral workflows would be able to read the
templates
out, modify them, and save back to swift.

Correct, but from the Mistral usage standpoint, having the
flexibility
is a good thing IMO regardless of the example I have chosen.

Object stores FTW

I Mistral action able to call either OpenStack service python
client or
OpenStack service API directly?

What is the response from the Mistral action in the workflow?
Lets say
we'd use Mistral to get a list of available environments (we do
this in
tripleo-common now) So we call Mistral API to trigger a
workflow that
has single action which gets the list of environments. Is
mistral able
to provide this list as a response, or it is able just to
trigger a
workflow?

In similar manner, is Mistral able to provide a way to get
workflow in
progress data? E.g. We have a Mistral workflow for nodes
introspection.
This workflow contains several actions calling to ironic and
ironic-inspector apis. GUI calls Mistral API to trigger the
workflow and
now it needs to have a way to track the progress of that
workflow. How
would this be achieved? I think forcing GUI to poll the APIs
that are
called in the actions and try to implement logic that estimates
what
state the workflow is to report about it is not a valid
solution. We
need the workflow API (Mistral) to provide a lets say web
sockets
connection and push the status of actions along with relevant
data, so
GUI can listen to those.

I can see a few options for progress polling/push.

Mistral:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Poll mistral to see how many are done

TripleO API
1. Build the REST API (we build)
1. Send requests to start introspection (we build)
1. Poll TripleO API until things are done

Mistral + Zaqar:
1. Send job to REST API (we didn't have to build)
1. It spins off as many jobs as it needs (we build)
1. Send updates to Zaqar (we didn't have to build)
1. Get websocket updates to the GUI from Zaqar (we didn't have to
build)

Zaqar is (of course) designed to deliver updates like this, so
every
project on the face of the planet doesn't have to rebuild
websocket
notifications, which is a good thing.

Thanks, this sounds very good. So the Mistral + Zaqar workflow
means
that as part of workflow action, Mistral notifies Zaqar that
certain
thing happened. Zaqar then notifies listening GUI providing
information
to identify what API call the GUI needs to make to retrieve
relevant
data or even provide the data itself (e.g. Mistral fails, some
error
occurs etc.)

I don't know what the mistral-zaqar integration is at the moment, but
if 
it's not there you could post to zaqar in the custom TripleO actions.

Exactly, This is what I had in mind as well. And if zaqarclient is in
good shape integrating it w/ Mistral should be quite straightforward.

I am about to implement your nodes workflow in the GUI to test
how it
works.

Jirka



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev

Jirka



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Dan_Prince (8,160 points)   1 5 8
0 votes

On 01/11/2016 11:09 PM, Tzu-Mainn Chen wrote:

----- Original Message -----

Background info:

We've got a problem in TripleO at the moment where many of our
workflows can be driven by the command line only. This causes some
problems for those trying to build a UI around the workflows in that
they have to duplicate deployment logic in potentially multiple places.
There are specs up for review which outline how we might solve this
problem by building what is called TripleO API [1].

Late last year I began experimenting with an OpenStack service called
Mistral which contains a generic workflow API. Mistral supports
defining workflows in YAML and then creating, managing, and executing
them via an OpenStack API. Initially the effort was focused around the
idea of creating a workflow in Mistral which could supplant our
"baremetal introspection" workflow which currently lives in python-
tripleoclient. I create a video presentation which outlines this effort
[2]. This particular workflow seemed to fit nicely within the Mistral
tooling.


More recently I've turned my attention to what it might look like if we
were to use Mistral as a replacement for the TripleO API entirely. This
brings forth the question of would TripleO be better off building out
its own API... or would relying on existing OpenStack APIs be a better
solution?

Some things I like about the Mistral solution:

  • The API already exists and is generic.

  • Mistral already supports interacting with many of the OpenStack API's
    we require [3]. Integration with keystone is baked in. Adding support
    for new clients seems straightforward (I've had no issues in adding
    support for ironic, inspector, and swift actions).

  • Mistral actions are pluggable. We could fairly easily wrap some of
    our more complex workflows (perhaps those that aren't easy to replicate
    with pure YAML workflows) by creating our own TripleO Mistral actions.
    This approach would be similar to creating a custom Heat resource...
    something we have avoided with Heat in TripleO but I think it is
    perhaps more reasonable with Mistral and would allow us to again build
    out our YAML workflows to drive things. This might allow us to build
    off some of the tripleo-common consolidation that is already underway
    ...

  • We could achieve a "stable API" by simply maintaining input
    parameters for workflows in a stable manner. Or perhaps workflows get
    versioned like a normal API would be as well.

  • The purist part of me likes Mistral quite a bit. It fits nicely with
    the deploy OpenStack with OpenStack. I sort of feel like if we have to
    build our own API in TripleO part of this vision has failed and could
    even be seen as a massive technical debt which would likely be hard to
    build a community around outside of TripleO.

  • Some of the proposed validations could perhaps be implemented as new
    Mistral actions as well. I'm not convinced we require TripleO API just
    to support a validations mechanism yet. Perhaps validations seem hard
    because we are simply trying to do them in the wrong places anyway?
    (like for example perhaps we should validate network connectivity at
    inspection time rather than during provisioning).

  • Power users might find a workflow built around a Mistral API more
    easy to interact with and expand upon. Perhaps this ends up being
    something that gets submitted as a patchset back to the TripleO that we
    accept into our upstream "stock" workflow sets.

Hiya! Thanks for putting down your thoughts.

I think I fundamentally disagree with the idea of using Mistral, simply
because many of the actions we'd like to expose through a REST API
(described in the tripleo-common deployment library spec [1]) aren't
workflows; they're straightforward get/set methods.

Right, because this spec describes nearly nothing from what is present
in tripleoclient now. And what we realistically have now is workflows,
which we'll have to reimplement in API somehow. So maybe we need both:
the get/set TripleO API for deployment plans and Mistral for workflows.

This would make sense to me. I have no objections to using Mistral for the
bits where we have actual workflows, only for the get/set-type methods
proposed in the spec. Using it for a latter seems like a major stretch,
as I argued in my previous email.

Mainn

Putting a workflow
engine in front of that feels like overkill and an added complication
that simply isn't needed. And added complications can lead to unneeded
complications: for instance, one open Mistral bug details how it may
not scale well [2].

Let's not talk about scaling in the context of what we have in
tripleoclient now ;)

The Mistral solution feels like we're trying to force a circular peg
into a round-ish hole. In a vacuum, if we were to consider the
engineering problem of exposing a code base to outside consumers in a
non-language specific fashion - I'm pretty sure we'd just suggest the
creation of a REST API and be done with it; the thought of using a
workflow engine as the frontend would not cross our minds.

I don't really agree with the 'purist' argument. We already have custom
business logic written in the TripleO CLI; accepting that within TripleO,
but not a very thin API layer, feels like an arbitrary line to me. And
if that line exists, I'd argue that if it forces overcomplicated
solutions to straightforward engineering problems, then that line needs
to be moved.

Mainn

[1]
https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst
[2] https://bugs.launchpad.net/mistral/+bug/1423054


Last week we landed the last patches [4] to our undercloud to enable
installing Mistral by simply setting: enable_mistral = true in
undercloud.conf. NOTE: you'll need to be using a recent trunk repo from
Delorean so that you have the recently added Mistral packages for this
to work. Although the feature is disable by default this should enable
those wishing to tinker with Mistral as a new TripleO undercloud
service an easy path forwards.

[1] https://review.openstack.org/#/c/230432
[2] https://www.youtube.com/watch?v=bnAT37O-sdw
[3] http://git.openstack.org/cgit/openstack/mistral/tree/mistral/action
s/openstack/mapping.json
[4] https://etherpad.openstack.org/p/tripleo-undercloud-workflow


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 12, 2016 by Tzu-Mainn_Chen (1,420 points)   4
...