settingsLogin | Registersettings

[openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

0 votes

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
    - derive_params:
          workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 14, 2017 in openstack-dev by Steven_Hardy (16,900 points)   2 7 11

9 Responses

0 votes

On 11/8/17 6:09 AM, Steven Hardy wrote:
Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

  pre_deploy_workflows:
     - derive_params:
           workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

I apologise for a generic/non-techy comment: it would be nice to keep
required workflows near the services' definition templates, to keep it
as much self-contained as possible. IIUC, that's covered by #1.
For future steps, I'd like to see all of the "bulk processing" to sit in
those templates as well.

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

Thanks Steven for the update.

Current CLI flow:


* User need to add -p parameter for the overcloud deploy command with
workflows to be invoked [1]
* Plan will update updated to the swift container
* Derived parameters workflow is initiated
- For each role
* Get the introspection data of first node assigned to the role
* Find the list features based on the services or parameters
* If dpdk present, run dpdk formulas workflow
* if sriov is present, run sriov formulas workfow (under development)
* if sriov or dpdk is present, run host formulas workflow
* if hci present, run hci formulas workflow

Here the order of the formulas workflow invocation is important. For
example, in Compute-DPDK-HCI role, HCI formulas should exclude the
CPUs allocated for DPDK PMD threads, while calculating cpu allocation
ratio.

I am trying to understand the proposed changes. Is it for assisting UI
only or changing the existing CLI flow too? If the idea is to invoke
the individual formulas workflow, it will not be possible with
existing implementation, need to be re-worked. We need to introduce
order for formulas workflow and direct fetching and merging of derived
parameters in plan.

As per earlier discussion jtomasek, to invoke derived parameters
workflow (existing) for a plan, UI requires following information:
* Whether derived parameters should be invoked for this deployment
(based on roles and enabled services)
* If yes, list of parameters, its types, and its default values (and
choices if present), are required

Did I miss anything?

Regards,
Saravanan KR

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/plan-samples/plan-environment-derived-params.yaml

On Wed, Nov 8, 2017 at 2:39 PM, Bogdan Dobrelya bdobreli@redhat.com wrote:
On 11/8/17 6:09 AM, Steven Hardy wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

  pre_deploy_workflows:
     - derive_params:
           workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

I apologise for a generic/non-techy comment: it would be nice to keep
required workflows near the services' definition templates, to keep it as
much self-contained as possible. IIUC, that's covered by #1.
For future steps, I'd like to see all of the "bulk processing" to sit in
those templates as well.

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Saravanan_KR (1,420 points)   5
0 votes

On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy shardy@redhat.com wrote:
Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
    - derive_params:
          workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

Instead of using mistral to express the pre_deploy workflows could we
just use pure ansible tasks/playbooks? We already use pure ansible for
upgrade and deploy, it seems disjointed to then require Mistral for
pre-deploy.

I think we should continue to move in the direction that the output
from Heat is ansible, which can then be driven via either Mistral or
ad-hoc via ansible-playbook. That pattern is more aligned with the
existing work around config-download as well. So, we would have a
Mistral workflow that automates what you describe above, except for
the detail that predeployworkflows becomes either predeploytasks
or predeployplaybooks, and the output of the Heat validation step is
pure ansible.

From the UI's perspective, whether the output from Heat is pure
ansible or mistral workflows should be an implementation detail hidden
behind a higher level mistral workflow that automates the full
"generation" of all pre_deploy stuff.

Also keep in mind Emilien's proposal to switch all scenario testing to
be driven via undercloud deploy. In that environment, there is no
Mistral, and everything is driven via config-download. What you're
proposing (unless I misunderstand) creates a tight coupling between
the Heat/Mistral ordering and requires one to use Mistral as part of
the deployment. Right now, we have no such requirement, and are able
to drive a deployment with only an ephemeral Heat and
ansible-playbook, which is the basis of how undercloud deploy works.

--
-- James Slagle
--


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by James_Slagle (7,000 points)   1 3 3
0 votes

On Wed, Nov 8, 2017 at 11:09 AM, Saravanan KR skramaja@redhat.com wrote:

Thanks Steven for the update.

Current CLI flow:


* User need to add -p parameter for the overcloud deploy command with
workflows to be invoked [1]
* Plan will update updated to the swift container
* Derived parameters workflow is initiated
- For each role
* Get the introspection data of first node assigned to the role
* Find the list features based on the services or parameters
* If dpdk present, run dpdk formulas workflow
* if sriov is present, run sriov formulas workfow (under
development)
* if sriov or dpdk is present, run host formulas workflow
* if hci present, run hci formulas workflow

Here the order of the formulas workflow invocation is important. For
example, in Compute-DPDK-HCI role, HCI formulas should exclude the
CPUs allocated for DPDK PMD threads, while calculating cpu allocation
ratio.

I am trying to understand the proposed changes. Is it for assisting UI
only or changing the existing CLI flow too? If the idea is to invoke
the individual formulas workflow, it will not be possible with
existing implementation, need to be re-worked. We need to introduce
order for formulas workflow and direct fetching and merging of derived
parameters in plan.

So there are several problems we're trying to solve with this proposal. In
general is to provide feature based workflows which will configure these
features, as well as provide means to get current configuration of these
features and provide sensible information about the input for these
workflows.

I think one of the main problem of current implementation is that user is
not able to get any information about input required to provide to run
derivation workflows. That information is purely documentation based and
also involves tweaking deployment-plan which I am convinced is not a good
way to provide the input.

So what we're proposing is to bring up a mechanism of mapping the
derivation workflows to services (roles/environments) so as Steven
described we're able to identify which workflows are possible to run and
provide extensive input definition so user can see what he is configuring
and why (input type, description, label).

This also means that there is several parameter derivation workflows rather
than just one and the input for the workflow is the actual input passed to
mistral (no plan-environment.yaml changes involved). Using this whole
approach means that for each such identified feature, we can provide -
Input details, general feature description (mistral workflow description)
and current configuration (by reaching to mistral workflow execution if
that was run before)

If as you're saying certain workflows depend on each other those should
probably be in one workflow. On the other hand, I think it is not goo
approach to try to put all the parameter derivation workflows into single
workflow.

-- Jirka

As per earlier discussion jtomasek, to invoke derived parameters
workflow (existing) for a plan, UI requires following information:
* Whether derived parameters should be invoked for this deployment
(based on roles and enabled services)
* If yes, list of parameters, its types, and its default values (and
choices if present), are required

Did I miss anything?

Regards,
Saravanan KR

[1] https://github.com/openstack/tripleo-heat-templates/blob/
master/plan-samples/plan-environment-derived-params.yaml

On Wed, Nov 8, 2017 at 2:39 PM, Bogdan Dobrelya bdobreli@redhat.com
wrote:

On 11/8/17 6:09 AM, Steven Hardy wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

  pre_deploy_workflows:
     - derive_params:
           workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

I apologise for a generic/non-techy comment: it would be nice to keep
required workflows near the services' definition templates, to keep it as
much self-contained as possible. IIUC, that's covered by #1.
For future steps, I'd like to see all of the "bulk processing" to sit in
those templates as well.

Thanks,

Steve



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Jiri_Tomasek (2,240 points)   2 3
0 votes

On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy shardy@redhat.com wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
    - derive_params:
          workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

As I think about this more, we may find out that matching a service to
workflow is not enough as workflow may require several services (together
defining a feature) So maybe doing it in separate file would help. E.g.

pre-deploy-workflows.yaml
- name: my.workflow
services: a, b, c, d

Maybe there is a better way, maybe this is not even needed. I am not sure.
What do you think?

What I really like about this proposal is that it provides a standard way
to configure deployment features and provides clear means to add additional
such configurations.

The resulting deployment configuration steps in GUI would look following:

1/ Hardware (reg. nodes, introspect etc)

2/ High level deployment configuration (basically selecting additional
environment files)

3/ Roles management (Roles selection, roles -> nodes assignment, roles
configuration - setting roles_data properties)

4/ Network configuration - network configuration wizard: (I'll describe
this in separate email)

5/ Deployment Features configuration (This proposal) - a list of features
to configure, the list is nicely generated from information provided in
previous steps, user has all the information to configure those features at
hand and can go through these step by step.

6/ Advanced deployment config - a view providing a way to review
Environment/Roles/Services parameters, search and tweak them if needed.

7/ Deploy.

I believe these steps should cover anything we should need to do for
deployment configuration.

-- Jirka

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Jiri_Tomasek (2,240 points)   2 3
0 votes

On Wed, Nov 8, 2017 at 7:16 AM, James Slagle james.slagle@gmail.com wrote:
On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy shardy@redhat.com wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

After chatting with jtomasek on irc, I wanted to clarify that the part
of this proposal I'm hesitant about.

Specifically, it's adding an interface for any service to specify a
mistral workflow, that in theory could do anything, as part of a
pre_deploy interface "contract". If we do offer such an interface, I
think it ought to be driven via ansible tasks/plays that are available
as Heat stack outputs to match the config-download pattern.

Perhaps for just deriving parameters, having a way to specify
workflows for the UI is Ok. It's more of the generic interface I'm not
so keen on. As it relates to your example of downloading container
images, it seems we could have a generic ansible task to do that, that
could then be executed with Mistral for API purposes instead of
specifying the Mistral workflow directly in the templates/roles_data.

--
-- James Slagle
--


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by James_Slagle (7,000 points)   1 3 3
0 votes

On Wed, Nov 8, 2017 at 10:55 PM, James Slagle james.slagle@gmail.com wrote:
On Wed, Nov 8, 2017 at 7:16 AM, James Slagle james.slagle@gmail.com wrote:

On Wed, Nov 8, 2017 at 12:09 AM, Steven Hardy shardy@redhat.com wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

After chatting with jtomasek on irc, I wanted to clarify that the part
of this proposal I'm hesitant about.

Specifically, it's adding an interface for any service to specify a
mistral workflow, that in theory could do anything, as part of a
pre_deploy interface "contract". If we do offer such an interface, I
think it ought to be driven via ansible tasks/plays that are available
as Heat stack outputs to match the config-download pattern.

Thanks for the feedback, yes I agree if we can do this with pure
ansible that would be great, but we'll have to take a closer look at
the existing implementation, e.g as mentioned by Saravanan there is an
existing integration with mistral workflows, which we'll either have
to maintain or migrate away from.

Perhaps for just deriving parameters, having a way to specify
workflows for the UI is Ok. It's more of the generic interface I'm not
so keen on. As it relates to your example of downloading container
images, it seems we could have a generic ansible task to do that, that
could then be executed with Mistral for API purposes instead of
specifying the Mistral workflow directly in the templates/roles_data.

Yeah good point, and also the point about CI moving towards undercloud
deploy is a good one - if we can work out a way to do this via ansible
(even if that means ansible running a mistral workflow as a
transitional step?) that would certainly be easier.

Hopefully we can chat more about this on IRC next week and prototype
the ansible approach to see how it could work.

Thanks!

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 9, 2017 by Steven_Hardy (16,900 points)   2 7 11
0 votes

On Thu, Nov 9, 2017 at 2:57 AM, Jiri Tomasek jtomasek@redhat.com wrote:

On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy shardy@redhat.com wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
    - derive_params:
          workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

As I think about this more, we may find out that matching a service to
workflow is not enough as workflow may require several services (together
defining a feature) So maybe doing it in separate file would help. E.g.

pre-deploy-workflows.yaml
- name: my.workflow
services: a, b, c, d

Maybe there is a better way, maybe this is not even needed. I am not sure.
What do you think?

Currently, HCI derive parameters workflow is invoked if a role has
both NovaCompute and CephOSD services enabled.

What I really like about this proposal is that it provides a standard way to
configure deployment features and provides clear means to add additional
such configurations.

The resulting deployment configuration steps in GUI would look following:

1/ Hardware (reg. nodes, introspect etc)

2/ High level deployment configuration (basically selecting additional
environment files)

3/ Roles management (Roles selection, roles -> nodes assignment, roles
configuration - setting roles_data properties)

4/ Network configuration - network configuration wizard: (I'll describe
this in separate email)

5/ Deployment Features configuration (This proposal) - a list of features to
configure, the list is nicely generated from information provided in
previous steps, user has all the information to configure those features at
hand and can go through these step by step.

Agreed on the UI workflow.

For DPDK and SR-IOV, there are common host specific parameters to be
derived. It has been added as a separate host-specific parameters
workflow. And both DPDK and SR-IOV workflow execution should follow
host-specific workflow.
In case of DPDK and HCI in same role, it is expected that DPDK
workflow is executed before HCI. And the service configuration should
provide this order to UI.
I am not able to realize how this information will be provided and
processed in UI with user. Do you have a UI wire frame for this
workflow?

6/ Advanced deployment config - a view providing a way to review
Environment/Roles/Services parameters, search and tweak them if needed.

7/ Deploy.

I believe these steps should cover anything we should need to do for
deployment configuration.

-- Jirka

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?
Agreed.

Regards,
Saravanan KR

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 9, 2017 by Saravanan_KR (1,420 points)   5
0 votes

As discussed in IRC, I have collated all the important discussions to
the etherpad (gdoc was not publicly shareable).

https://etherpad.openstack.org/p/tripleo-derive-parameters-v2

Lets continue discussion on the etherpad to finalize.

Regards,
Saravanan KR

On Thu, Nov 9, 2017 at 11:05 AM, Saravanan KR skramaja@redhat.com wrote:
On Thu, Nov 9, 2017 at 2:57 AM, Jiri Tomasek jtomasek@redhat.com wrote:

On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy shardy@redhat.com wrote:

Hi all,

Today I had a productive hallway discussion with jtomasek and
stevebaker re $subject, so I wanted to elaborate here for the benefit
of those folks not present. Hopefully we can get feedback on the
ideas and see if it makes sense to continue and work on some patches:

The problem under discussion is how do we run pre-deployment workflows
(such as those integrated recently to calculate derived parameters,
and in future perhaps also those which download container images etc),
and in particular how do we make these discoverable via the UI
(including any input parameters).

The idea we came up with has two parts:

  1. Add a new optional section to roles_data for services that require
    pre-deploy workflows

E.g something like this:

 pre_deploy_workflows:
    - derive_params:
          workflow_name:

tripleo.deriveparamsformulas.v1.dpdkderiveparams
inputs:
...

This would allow us to associate a specific mistral workflow with a
given service template, and also work around the fact that currently
mistral inputs don't have any schema (only key/value input) as we
could encode the required type and any constraints in the inputs block
(clearly this could be removed in future should typed parameters
become available in mistral).

  1. Add a new workflow that calculates the enabled services and returns
    all predeployworkflows

This would take all enabled environments, then use heat to validate
the configuration and return the merged resource registry (which will
require https://review.openstack.org/#/c/509760/), then we would
iterate over all enabled services in the registry and extract a given
rolesdata key (e.g predeploy_workflows)

The result of the workflow would be a list of all predeployworkflows
for all enabled services, which the UI could then use to run the
workflows as part of the pre-deploy process.

As I think about this more, we may find out that matching a service to
workflow is not enough as workflow may require several services (together
defining a feature) So maybe doing it in separate file would help. E.g.

pre-deploy-workflows.yaml
- name: my.workflow
services: a, b, c, d

Maybe there is a better way, maybe this is not even needed. I am not sure.
What do you think?

Currently, HCI derive parameters workflow is invoked if a role has
both NovaCompute and CephOSD services enabled.

What I really like about this proposal is that it provides a standard way to
configure deployment features and provides clear means to add additional
such configurations.

The resulting deployment configuration steps in GUI would look following:

1/ Hardware (reg. nodes, introspect etc)

2/ High level deployment configuration (basically selecting additional
environment files)

3/ Roles management (Roles selection, roles -> nodes assignment, roles
configuration - setting roles_data properties)

4/ Network configuration - network configuration wizard: (I'll describe
this in separate email)

5/ Deployment Features configuration (This proposal) - a list of features to
configure, the list is nicely generated from information provided in
previous steps, user has all the information to configure those features at
hand and can go through these step by step.

Agreed on the UI workflow.

For DPDK and SR-IOV, there are common host specific parameters to be
derived. It has been added as a separate host-specific parameters
workflow. And both DPDK and SR-IOV workflow execution should follow
host-specific workflow.
In case of DPDK and HCI in same role, it is expected that DPDK
workflow is executed before HCI. And the service configuration should
provide this order to UI.
I am not able to realize how this information will be provided and
processed in UI with user. Do you have a UI wire frame for this
workflow?

6/ Advanced deployment config - a view providing a way to review
Environment/Roles/Services parameters, search and tweak them if needed.

7/ Deploy.

I believe these steps should cover anything we should need to do for
deployment configuration.

-- Jirka

If this makes sense I can go ahead and push some patches so we can
iterate on the implementation?
Agreed.

Regards,
Saravanan KR

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Saravanan_KR (1,420 points)   5
...