settingsLogin | Registersettings

[openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of Heat, containers, and the future of TripleO

0 votes

tl;dr Containers represent a massive, and also mandatory, opportunity
for TripleO. Lets start thinking about ways that we can take maximum
advantage to achieve the goals of the project.

Now that you have the tl;dr I'm going to start from the beginning, so
settle in and grab yourself a cup of coffee or other poison of your choice.

After working on developing Heat from the very beginning of the project
in early 2012 and debugging a bunch of TripleO deployments in the field,
it is my considered opinion that Heat is a poor fit for the workloads
that TripleO is currently asking of it. To illustrate why, I need to
explain what it is that Heat is really designed to do.

Here's a theoretical example of how I've always imagined Heat software
deployments would make Heat users' lives better. For simplicity, I'm
just going to model two software components, a user-facing service that
connects to some back-end service:

resources:
backendcomponent:
type: OS::Heat::SoftwareComponent
properties:
configs:
- tool: script
actions:
- CREATE
- UPDATE
config: |
PORT=$(get
backendport || randomport)
stopbackend
start
backend $DEPLOYVERSION $PORT $CONFIG
addr="$(hostname):$(get
backendport)"
printf '%s' "$addr" >${heat
outputspath}.hostandport
- tool: script
actions:
- DELETE
config: |
stop
backend
inputs:
- name: DEPLOYVERSION
- name: CONFIG
outputs:
- name: host
and_port

 frontend_component:
   type: OS::Heat::SoftwareComponent
   properties:
     configs:
       - tool: script
         actions:
           - CREATE
           - UPDATE
         config: |
           stop_frontend
           start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG
       - tool: script
         actions:
           - DELETE
         config: |
           stop_frontend
     inputs:
       - name: DEPLOY_VERSION
       - name: BACKEND_ADDR
       - name: CONFIG

 backend:
   type: OS::Heat::SoftwareDeployment
   properties:
     server: {get_resource: backend_server}
     name: {get_param: backend_version} # Forces upgrade replacement
     actions: [CREATE, UPDATE, DELETE]
     config: {get_resource: backend_component}
     input_values:
       DEPLOY_VERSION: ${get_param: backend_version}
       CONFIG: ${get_param: backend_config}

 frontend:
   type: OS::Heat::SoftwareDeployment
   properties:
     server: {get_resource: frontend_server}
     name: {get_param: frontend_version} # Forces upgrade replacement
     actions: [CREATE, UPDATE, DELETE]
     config: {get_resource: frontend_component}
     input_values:
       DEPLOY_VERSION: ${get_param: frontend_version}
       BACKEND_ADDR: {get_attr: [backend, host_and_port]}
       CONFIG: ${get_param: frontend_config}

This is actually quite a beautiful system, if I may say so:

  • Whenever a version changes, Heat knows to update that component, and
    the components can be updated independently.
  • If the backend in this example restarts on a different port, the
    frontend is updated to point to the new port.
  • Everything is completely agnostic as to which server it is running on.
    They could be running on the same server or different servers.
  • Everything is integrated with the infrastructure (not only the servers
    you're deploying on and the networks and volumes connected to them, but
    also things like load balancers), so everything is created at the right
    time, in parallel where possible, and any errors are reported all in one
    place.
  • If something requires e.g. a restart after changing another component,
    we can encode that. And if it doesn't, we can encode that too.
  • There's next to no downtime required: if e.g. we upgrade the backend,
    we first deploy a new one listening on a new port, then update the
    frontend to listen on the new port, then finally shut down the old
    backend. Again, we can choose when we want this and when we just want to
    update in place and reload.
  • The application doesn't even need to worry about versioning the
    protocol that its two constituent parts communicate over: as long as the
    backendversion and frontendversion that we pass are always compatible,
    only compatible versions of the two services ever talk to each other.
  • If anything at all fails at any point before, during or after this
    part of the template, Heat can automatically roll everything back into
    the exact same state as it was in before, without any outside
    intervention. You can insert test deployments that check everything is
    working and have them automatically roll back if it's not, all with no
    downtime for users.

So you can use this to do something like a fancier version of blue-green
deployment,[1] where you're actually rolling out the (virtualised)
hardware and infrastructure in a blue-green fashion along with the
software. Not only that, you can choose to replace your whole stack or
only parts of it. (Note: the way I had to encode this in the example
above, by changing the deployment name so that it forces a resource
replacement, is a hack. We really need a feature to specify in a
software config resource which inputs should result in a replacement on
change.)

It's worth noting that in practice you really, really want everything
deployed in containers to make this process work consistently, even
though in theory you could make this work (briefly) without them. In
particular, rollback without containers is a dicey proposition. When we
first started talking about implementing software deployments in Heat I
half-seriously suggested that maybe we should make containers the only
allowed type of software deployment, and I kind of wonder now if I
shouldn't have pressed harder on that point.

In any event, unfortunately as everyone involved in TripleO knows, the
way TripleO uses Heat looks nothing like this. It actually looks more
like this:

resources:
installallthethingsononeserverconfig:
type: OS::Heat::SoftwareConfig
properties:
actions: [CREATE]
config: {get
file: installallthethingsononeserver.sh}

 update_all_the_things_on_one_server_config:
   type: OS::Heat::SoftwareConfig
   properties:
     actions: [UPDATE]
     config: {get_file: update_all_the_things_on_one_server.sh}
     inputs:
       - name: update_count

 ...

(Filling in the rest is left as an exercise to the reader. You're welcome.)

Not illustrated are the multiple sources of truth that we have: puppet
modules (packaged on the server), puppet manifests and hieradata
(delivered via Heat), external package repositories. Heat is a dataflow
language but much of the data it should be operating on is actually
hidden from it. That's going about as well as you might expect.

Due to the impossibility of ever rolling back a deployment like one of
those, we just disable rollback for the overcloud templates, so if
there's a failure we end up stuck in whatever intermediate state we were
in when the script died. That can leave things in an state where
recovery is not automatic when 'earlier' deployments (like the package
update) end up depending on state set up by 'later' deployments (like
the post- scripts, which manipulate Pacemaker's state in Pacemaker-based
deployments). Even worse, many of the current scripts leave the machine
in a state that requires manual recovery should they fail part-way through.

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.

So what would be a good alternative? And how would we evaluate the options?

To my mind, the purpose of the TripleO project is this: to ensure that
there is an OpenStack community collaborating around each part of the
OpenStack installation/management story. We don't care about TripleO
"owning" that part (all things being equal, we'd prefer not to), just
that nobody should have to go outside the OpenStack community and/or
roll their own thing to install OpenStack unless they want to. So I
think the ability to sustain a community around whatever solution we
choose ought to be a primary consideration.

The use of Ironic has been something of a success story here. There's
only one place to add hardware support to enable both installing
OpenStack itself on bare-metal via TripleO and the 'regular'
bare-metal-to-tenant use case of Ironic. This is a clear win/win.

Beyond getting the bare-metal machines marshalled, the other part of the
solution is configuration management and orchestration of the various
software services. When TripleO started there was nowhere in OpenStack
that was defining the relationships between services needed to
orchestrate them. To a large extent there still isn't. I think that one
of the reasons we adopted Puppet in TripleO was that it was supposed to
provide this, at least within a limited scope (i.e. on one machine - the
puppet-deploying community is largely using Ansible to orchestrate
across boxes, and we are using Heat). However, what we've discovered in
the past few months is that Puppet is actually not able to fulfil this
role as long as we support Pacemaker-based deployments as an option,
because in that case Pacemaker actually has control of starting and
stopping all of the services. As a result we are back to defining it all
ourselves in the Pacemaker config plus various hacky shell scripts,
instead of relying on (and contributing to!) a larger community. Even
ignoring that, Puppet doesn't solve the problem of orchestrating across
multiple machines.

Clearly one option would be to encode everything in Heat along the lines
of the first example above. I think once we have containers this could
actually work really well for compute nodes and other types of scale-out
nodes (e.g. Swift nodes). The scale-out model of Heat scaling groups
works really well for this use case, and between the improvements we
have put in place (like batched updates and user hooks) and those still
on the agenda (like notifications + automatic Mistral workflow
triggering on hooks) Heat could provide a really good way of capturing
things like migrating user workloads on scale down and rolling updates
in the templates, so that they can be managed completely automatically
by the undercloud with no client involvement (and when the undercloud
becomes HA, they'll get HA for free). I'd be pretty excited to see this
tried. The potential downside is that the orchestration definitions are
still trapped inside the TripleO templates, so they're not being shared
outside of the TripleO community. This is probably justified though
owing to its close ties to the underlying infrastructure.

An alternative out of left field: as far as I can gather the "completely
new way of orchestrating activities" used by the new Puppet Application
Orchestration thing[2] uses substantially the same model as I described
for Heat above. If we added Puppet Application Orchestration data to
openstack-puppet-modules then it may be possible to write a tool to
generate Heat templates from that data. However in talking with Emilien
it sounds like o-p-m is quite some time away from tackling PAO. So I
don't think this is really feasible.

In any event, it's when we get to the controller nodes that the
downsides become more pronounced. We're no longer talking about one
deployment per service like I sketched above; each service is actually
multiple deployments forming an active-active cluster with virtual IPs
and failover and all that jazz. It may be that everything would just
work the same way, but we would be in uncharted territory and there
would likely be unanticipated subtleties. It's particularly unclear how
we would handle stop-the-world database migrations in this model,
although we do have the option of hoping that stop-the-world database
migrations will have been completely phased out by then.

To make it even more complicated, we ultimately want the services to
heterogeneously spread among controller nodes in a configurable way. I
believe that Dan's work on composable roles has already gone some way
toward this without even using containers, but it's likely to become
increasingly difficult to model in Heat without some sort of template
generation. (I personally think that template generation would be a Good
Thing, but we've chosen not to go down that path so far.) Quite possibly
even just having composable roles could make it untenable to continue
maintaining separate Pacemaker and non-Pacemaker deployment modes. It'd
be really nice to have the flexibility to do things like scale out
different services at different rates. What's more, we are going to need
some way of redistributing services when a machine in the cluster fails,
and ultimately we would like that process to be automated, which would
require a template generation service.

We certainly could build all of that. But we definitely shouldn't
because this is the kind of thing that services like Kubernetes and
Apache Mesos are designed to do already. And that raises another
possibility: Angus & friends are working on capturing the orchestration
relationships for Mesos+Marathon within the Kolla project (specifically,
in the kolla-mesos repository). This represents a tremendous opportunity
for the TripleO project to further its mission of having the same
deployment tools available to everyone as an official part of the
OpenStack project without having to maintain them separately.

As of the Liberty release, Magnum now supports provisioning Mesos
clusters, so TripleO wouldn't have to maintain the installer for that
either. (The choice of Mesos is somewhat unfortunate in our case,
because Magnum's Kubernetes support is much more mature than its Mesos
support, and because the reasons for the decision are about to be or
have already been overtaken by events - I've heard reports that the
features that Kubernetes was missing to allow it to be used for
controller nodes, and maybe even compute nodes, are now available.
Nonetheless, I expect the level of Magnum support for Mesos is likely
workable.) This is where the TripleO strategy of using OpenStack to
deploy OpenStack can really pay dividends: because we use Ironic all of
our servers are accessible through the Nova API, so in theory we can
just run Magnum out of the box.

The chances of me personally having time to prototype this are
slim-to-zero, but I think this is a path worth investigating.

cheers,
Zane.

[1] http://martinfowler.com/bliki/BlueGreenDeployment.html
[2] https://puppetlabs.com/introducing-puppet-application-orchestration


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 21, 2016 in openstack-dev by Zane_Bitter (21,640 points)   4 6 9

36 Responses

0 votes

This is quite a long mail, I'll reply to some statements inline.

On Mon, Mar 21, 2016 at 4:14 PM, Zane Bitter zbitter@redhat.com wrote:
tl;dr Containers represent a massive, and also mandatory, opportunity for
TripleO. Lets start thinking about ways that we can take maximum advantage
to achieve the goals of the project.

Now that you have the tl;dr I'm going to start from the beginning, so settle
in and grab yourself a cup of coffee or other poison of your choice.

After working on developing Heat from the very beginning of the project in
early 2012 and debugging a bunch of TripleO deployments in the field, it is
my considered opinion that Heat is a poor fit for the workloads that TripleO
is currently asking of it. To illustrate why, I need to explain what it is
that Heat is really designed to do.

Here's a theoretical example of how I've always imagined Heat software
deployments would make Heat users' lives better. For simplicity, I'm just
going to model two software components, a user-facing service that connects
to some back-end service:

resources:
backendcomponent:
type: OS::Heat::SoftwareComponent
properties:
configs:
- tool: script
actions:
- CREATE
- UPDATE
config: |
PORT=$(get
backendport || randomport)
stopbackend
start
backend $DEPLOYVERSION $PORT $CONFIG
addr="$(hostname):$(get
backendport)"
printf '%s' "$addr" >${heat
outputspath}.hostandport
- tool: script
actions:
- DELETE
config: |
stop
backend
inputs:
- name: DEPLOYVERSION
- name: CONFIG
outputs:
- name: host
and_port

frontend_component:
  type: OS::Heat::SoftwareComponent
  properties:
    configs:
      - tool: script
        actions:
          - CREATE
          - UPDATE
        config: |
          stop_frontend
          start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG
      - tool: script
        actions:
          - DELETE
        config: |
          stop_frontend
    inputs:
      - name: DEPLOY_VERSION
      - name: BACKEND_ADDR
      - name: CONFIG

backend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: backend_server}
    name: {get_param: backend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: backend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: backend_version}
      CONFIG: ${get_param: backend_config}

frontend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: frontend_server}
    name: {get_param: frontend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: frontend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: frontend_version}
      BACKEND_ADDR: {get_attr: [backend, host_and_port]}
      CONFIG: ${get_param: frontend_config}

This is actually quite a beautiful system, if I may say so:

  • Whenever a version changes, Heat knows to update that component, and the
    components can be updated independently.
  • If the backend in this example restarts on a different port, the frontend
    is updated to point to the new port.
  • Everything is completely agnostic as to which server it is running on.
    They could be running on the same server or different servers.
  • Everything is integrated with the infrastructure (not only the servers
    you're deploying on and the networks and volumes connected to them, but also
    things like load balancers), so everything is created at the right time, in
    parallel where possible, and any errors are reported all in one place.
  • If something requires e.g. a restart after changing another component, we
    can encode that. And if it doesn't, we can encode that too.
  • There's next to no downtime required: if e.g. we upgrade the backend, we
    first deploy a new one listening on a new port, then update the frontend to
    listen on the new port, then finally shut down the old backend. Again, we
    can choose when we want this and when we just want to update in place and
    reload.
  • The application doesn't even need to worry about versioning the protocol
    that its two constituent parts communicate over: as long as the
    backendversion and frontendversion that we pass are always compatible,
    only compatible versions of the two services ever talk to each other.
  • If anything at all fails at any point before, during or after this part of
    the template, Heat can automatically roll everything back into the exact
    same state as it was in before, without any outside intervention. You can
    insert test deployments that check everything is working and have them
    automatically roll back if it's not, all with no downtime for users.

So you can use this to do something like a fancier version of blue-green
deployment,[1] where you're actually rolling out the (virtualised) hardware
and infrastructure in a blue-green fashion along with the software. Not only
that, you can choose to replace your whole stack or only parts of it. (Note:
the way I had to encode this in the example above, by changing the
deployment name so that it forces a resource replacement, is a hack. We
really need a feature to specify in a software config resource which inputs
should result in a replacement on change.)

It's worth noting that in practice you really, really want everything
deployed in containers to make this process work consistently, even though
in theory you could make this work (briefly) without them. In particular,
rollback without containers is a dicey proposition. When we first started
talking about implementing software deployments in Heat I half-seriously
suggested that maybe we should make containers the only allowed type of
software deployment, and I kind of wonder now if I shouldn't have pressed
harder on that point.

In any event, unfortunately as everyone involved in TripleO knows, the way
TripleO uses Heat looks nothing like this. It actually looks more like this:

resources:
installallthethingsononeserverconfig:
type: OS::Heat::SoftwareConfig
properties:
actions: [CREATE]
config: {get
file: installallthethingsononeserver.sh}

update_all_the_things_on_one_server_config:
  type: OS::Heat::SoftwareConfig
  properties:
    actions: [UPDATE]
    config: {get_file: update_all_the_things_on_one_server.sh}
    inputs:
      - name: update_count

...

(Filling in the rest is left as an exercise to the reader. You're welcome.)

I have to be honest, I'm not OpenStack Heat expert and the last block
is very complex to understand.
Maybe a simple example with Keystone API would have been great.

Not illustrated are the multiple sources of truth that we have: puppet
modules (packaged on the server), puppet manifests and hieradata (delivered
via Heat), external package repositories. Heat is a dataflow language but
much of the data it should be operating on is actually hidden from it.
That's going about as well as you might expect.

Due to the impossibility of ever rolling back a deployment like one of
those, we just disable rollback for the overcloud templates, so if there's a
failure we end up stuck in whatever intermediate state we were in when the
script died. That can leave things in an state where recovery is not
automatic when 'earlier' deployments (like the package update) end up
depending on state set up by 'later' deployments (like the post- scripts,
which manipulate Pacemaker's state in Pacemaker-based deployments). Even
worse, many of the current scripts leave the machine in a state that
requires manual recovery should they fail part-way through.

Indeed, this has literally none of the benefits of the ideal Heat deployment
enumerated above save one: it may be entirely the wrong tool in every way
for the job it's being asked to do, but at least it is still well-integrated
with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack', where we
have one stack for the infrastructure and a separate one for the software
deployments, so that there is no longer any tight integration between
infrastructure and software. Although it makes me a bit sad in some ways, I
can certainly appreciate the merits of the idea as well. However, from the
argument above we can deduce that if this is the only thing we do then we
will end up in the very worst of all possible worlds: the wrong tool for the
job, poorly integrated. Every single advantage of using Heat to deploy
software will have evaporated, leaving only disadvantages.

So what would be a good alternative? And how would we evaluate the options?

To my mind, the purpose of the TripleO project is this: to ensure that there
is an OpenStack community collaborating around each part of the OpenStack
installation/management story. We don't care about TripleO "owning" that
part (all things being equal, we'd prefer not to), just that nobody should
have to go outside the OpenStack community and/or roll their own thing to
install OpenStack unless they want to. So I think the ability to sustain a
community around whatever solution we choose ought to be a primary
consideration.

The use of Ironic has been something of a success story here. There's only
one place to add hardware support to enable both installing OpenStack itself
on bare-metal via TripleO and the 'regular' bare-metal-to-tenant use case of
Ironic. This is a clear win/win.

Beyond getting the bare-metal machines marshalled, the other part of the
solution is configuration management and orchestration of the various
software services. When TripleO started there was nowhere in OpenStack that
was defining the relationships between services needed to orchestrate them.
To a large extent there still isn't. I think that one of the reasons we
adopted Puppet in TripleO was that it was supposed to provide this, at least
within a limited scope (i.e. on one machine - the puppet-deploying community
is largely using Ansible to orchestrate across boxes, and we are using
Heat). However, what we've discovered in the past few months is that Puppet
is actually not able to fulfil this role as long as we support
Pacemaker-based deployments as an option, because in that case Pacemaker
actually has control of starting and stopping all of the services. As a
result we are back to defining it all ourselves in the Pacemaker config plus
various hacky shell scripts, instead of relying on (and contributing to!) a
larger community. Even ignoring that, Puppet doesn't solve the problem of
orchestrating across multiple machines.

That is True there was/is an overlap between Puppet & Pacemaker
orchestration for services management, that's why we disable service
management in Puppet.
This is something we are working on by created a Puppet provider to
control 'pcs'.

https://review.openstack.org/#/c/286124/

Thanks to this work, deployments done by Puppet will manage resources
using pcs CLI and we'll be able to cleanup a lot of hacks that made it
possible in the past.
I agree Pacemaker is not helping, it has nothing to do with the
deployment tool: Pacemaker is complex to deploy and manage.
On a positive way, we are working on improving the Pacemaker/Puppet
collision and I think we will fix that during Newton cycle.

Clearly one option would be to encode everything in Heat along the lines of
the first example above. I think once we have containers this could actually
work really well for compute nodes and other types of scale-out nodes (e.g.
Swift nodes). The scale-out model of Heat scaling groups works really well
for this use case, and between the improvements we have put in place (like
batched updates and user hooks) and those still on the agenda (like
notifications + automatic Mistral workflow triggering on hooks) Heat could
provide a really good way of capturing things like migrating user workloads
on scale down and rolling updates in the templates, so that they can be
managed completely automatically by the undercloud with no client
involvement (and when the undercloud becomes HA, they'll get HA for free).
I'd be pretty excited to see this tried. The potential downside is that the
orchestration definitions are still trapped inside the TripleO templates, so
they're not being shared outside of the TripleO community. This is probably
justified though owing to its close ties to the underlying infrastructure.

An alternative out of left field: as far as I can gather the "completely new
way of orchestrating activities" used by the new Puppet Application
Orchestration thing[2] uses substantially the same model as I described for
Heat above. If we added Puppet Application Orchestration data to
openstack-puppet-modules then it may be possible to write a tool to generate
Heat templates from that data. However in talking with Emilien it sounds
like o-p-m is quite some time away from tackling PAO. So I don't think this
is really feasible.

That is True, Puppet OpenStack modules do not support PAO and I
haven't seen any efforts from Puppetlabs to help us doing that.
I'm even not sure about how it would work, since it require a
Puppetmaster and some pieces are not Open-Source. Beurk.

In any event, it's when we get to the controller nodes that the downsides
become more pronounced. We're no longer talking about one deployment per
service like I sketched above; each service is actually multiple deployments
forming an active-active cluster with virtual IPs and failover and all that
jazz. It may be that everything would just work the same way, but we would
be in uncharted territory and there would likely be unanticipated
subtleties. It's particularly unclear how we would handle stop-the-world
database migrations in this model, although we do have the option of hoping
that stop-the-world database migrations will have been completely phased out
by then.

To make it even more complicated, we ultimately want the services to
heterogeneously spread among controller nodes in a configurable way. I
believe that Dan's work on composable roles has already gone some way toward
this without even using containers, but it's likely to become increasingly
difficult to model in Heat without some sort of template generation. (I
personally think that template generation would be a Good Thing, but we've
chosen not to go down that path so far.) Quite possibly even just having
composable roles could make it untenable to continue maintaining separate
Pacemaker and non-Pacemaker deployment modes. It'd be really nice to have
the flexibility to do things like scale out different services at different
rates. What's more, we are going to need some way of redistributing services
when a machine in the cluster fails, and ultimately we would like that
process to be automated, which would require a template generation
service.

We certainly could build all of that. But we definitely shouldn't because
this is the kind of thing that services like Kubernetes and Apache Mesos are
designed to do already. And that raises another possibility: Angus & friends
are working on capturing the orchestration relationships for Mesos+Marathon
within the Kolla project (specifically, in the kolla-mesos repository). This
represents a tremendous opportunity for the TripleO project to further its
mission of having the same deployment tools available to everyone as an
official part of the OpenStack project without having to maintain them
separately.

As of the Liberty release, Magnum now supports provisioning Mesos clusters,
so TripleO wouldn't have to maintain the installer for that either. (The
choice of Mesos is somewhat unfortunate in our case, because Magnum's
Kubernetes support is much more mature than its Mesos support, and because
the reasons for the decision are about to be or have already been overtaken
by events - I've heard reports that the features that Kubernetes was missing
to allow it to be used for controller nodes, and maybe even compute nodes,
are now available. Nonetheless, I expect the level of Magnum support for
Mesos is likely workable.) This is where the TripleO strategy of using
OpenStack to deploy OpenStack can really pay dividends: because we use
Ironic all of our servers are accessible through the Nova API, so in theory
we can just run Magnum out of the box.

This is something we already & partially discussed when I submitted
the Containers blueprint, in September 2015:
https://review.openstack.org/#/c/222002/

I was told to abandon this blueprint, so Container folks could create a new one:
https://review.openstack.org/#/c/223182/

The "new" blueprint was never merged so I guess TripleO team accepted
the design, moreoever discussed during Tokyo Summit.
I am sad to see you were not involved in both blueprints, nor present
during our discussions, but let's move forward.

The chances of me personally having time to prototype this are slim-to-zero,
but I think this is a path worth investigating.

A lot of people are currently very busy, because of releases and other
critical topics, like CI, split stack and composable roles, maybe
people focusing on containers can also reply to this thread and try to
understand why we're here now.

I know it has currently no impact on TripleO, but I plan to gather
feedback about Puppet & Containers usage during next Summit in Austin,
I know TimeWarner is deploying OpenStack with docker & Puppet
OpenStack modules at scale, and they have a pretty nice experience on
that.
--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by emilien_at_redhat.co (36,940 points)   2 6 9
0 votes

+1 for TripleO taking a look at Kolla.

Some random thoughts:

I'm in the middle of deploying a new cloud and I couldn't use either TripleO or Kolla for various reasons. A few reasons for each:
* TripeO - worries me for ever having to do a major upgrade of the software, or needing to do oddball configs like vxlans over ipoib.
* Kolla - At the time it was still immature. No stable artefacts posted. database container recently broke, little documentation for disaster recovery. No upgrade strategy at the time.

Kolla rearchitected recently to support oddball configs like we've had to do at times. They also recently gained upgrade support. I think they are on the right path. If I had to start fresh, I'd very seriously consider using it.

I think Kolla can provide the missing pieces that TripleO needs. TripleO has bare metal deployment down solid. I really like the idea of using OpenStack to deploy OpenStack. Kolla is now OpenStack so should be considered.

I'm also in favor of using Magnum to deploy a COE to manage Kolla. I'm much less thrilled about Mesos though. It feels heavy enough weight that it feels like your deploying an OpenStack like system just to deploy OpenStack. So, OpenStack On NotOpenStack On OpenStack. :/ I've had good luck with Kubernetes (much simpler) recently and am disappointed that it was too immature at the time Kolla originally considered it. It seems much more feasible to use now. I use net=host like features all the time which was a major sticking point before.

I'd be interested in seeing TripeO use the Ansible version for now since that's working, stable, and supports upgrades/oddball configs. Then in the future as Kubernetes support or maybe Mesos support matures, consider that. Kolla's going to have to have a migration path from one to the other eventually... I think this would allow TripeO to really come into its own as an end to end, production ready system sooner.

Thanks,
Kevin


From: Emilien Macchi [emilien@redhat.com]
Sent: Tuesday, March 22, 2016 5:18 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of Heat, containers, and the future of TripleO

This is quite a long mail, I'll reply to some statements inline.

On Mon, Mar 21, 2016 at 4:14 PM, Zane Bitter zbitter@redhat.com wrote:
tl;dr Containers represent a massive, and also mandatory, opportunity for
TripleO. Lets start thinking about ways that we can take maximum advantage
to achieve the goals of the project.

Now that you have the tl;dr I'm going to start from the beginning, so settle
in and grab yourself a cup of coffee or other poison of your choice.

After working on developing Heat from the very beginning of the project in
early 2012 and debugging a bunch of TripleO deployments in the field, it is
my considered opinion that Heat is a poor fit for the workloads that TripleO
is currently asking of it. To illustrate why, I need to explain what it is
that Heat is really designed to do.

Here's a theoretical example of how I've always imagined Heat software
deployments would make Heat users' lives better. For simplicity, I'm just
going to model two software components, a user-facing service that connects
to some back-end service:

resources:
backendcomponent:
type: OS::Heat::SoftwareComponent
properties:
configs:
- tool: script
actions:
- CREATE
- UPDATE
config: |
PORT=$(get
backendport || randomport)
stopbackend
start
backend $DEPLOYVERSION $PORT $CONFIG
addr="$(hostname):$(get
backendport)"
printf '%s' "$addr" >${heat
outputspath}.hostandport
- tool: script
actions:
- DELETE
config: |
stop
backend
inputs:
- name: DEPLOYVERSION
- name: CONFIG
outputs:
- name: host
and_port

frontend_component:
  type: OS::Heat::SoftwareComponent
  properties:
    configs:
      - tool: script
        actions:
          - CREATE
          - UPDATE
        config: |
          stop_frontend
          start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG
      - tool: script
        actions:
          - DELETE
        config: |
          stop_frontend
    inputs:
      - name: DEPLOY_VERSION
      - name: BACKEND_ADDR
      - name: CONFIG

backend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: backend_server}
    name: {get_param: backend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: backend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: backend_version}
      CONFIG: ${get_param: backend_config}

frontend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: frontend_server}
    name: {get_param: frontend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: frontend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: frontend_version}
      BACKEND_ADDR: {get_attr: [backend, host_and_port]}
      CONFIG: ${get_param: frontend_config}

This is actually quite a beautiful system, if I may say so:

  • Whenever a version changes, Heat knows to update that component, and the
    components can be updated independently.
  • If the backend in this example restarts on a different port, the frontend
    is updated to point to the new port.
  • Everything is completely agnostic as to which server it is running on.
    They could be running on the same server or different servers.
  • Everything is integrated with the infrastructure (not only the servers
    you're deploying on and the networks and volumes connected to them, but also
    things like load balancers), so everything is created at the right time, in
    parallel where possible, and any errors are reported all in one place.
  • If something requires e.g. a restart after changing another component, we
    can encode that. And if it doesn't, we can encode that too.
  • There's next to no downtime required: if e.g. we upgrade the backend, we
    first deploy a new one listening on a new port, then update the frontend to
    listen on the new port, then finally shut down the old backend. Again, we
    can choose when we want this and when we just want to update in place and
    reload.
  • The application doesn't even need to worry about versioning the protocol
    that its two constituent parts communicate over: as long as the
    backendversion and frontendversion that we pass are always compatible,
    only compatible versions of the two services ever talk to each other.
  • If anything at all fails at any point before, during or after this part of
    the template, Heat can automatically roll everything back into the exact
    same state as it was in before, without any outside intervention. You can
    insert test deployments that check everything is working and have them
    automatically roll back if it's not, all with no downtime for users.

So you can use this to do something like a fancier version of blue-green
deployment,[1] where you're actually rolling out the (virtualised) hardware
and infrastructure in a blue-green fashion along with the software. Not only
that, you can choose to replace your whole stack or only parts of it. (Note:
the way I had to encode this in the example above, by changing the
deployment name so that it forces a resource replacement, is a hack. We
really need a feature to specify in a software config resource which inputs
should result in a replacement on change.)

It's worth noting that in practice you really, really want everything
deployed in containers to make this process work consistently, even though
in theory you could make this work (briefly) without them. In particular,
rollback without containers is a dicey proposition. When we first started
talking about implementing software deployments in Heat I half-seriously
suggested that maybe we should make containers the only allowed type of
software deployment, and I kind of wonder now if I shouldn't have pressed
harder on that point.

In any event, unfortunately as everyone involved in TripleO knows, the way
TripleO uses Heat looks nothing like this. It actually looks more like this:

resources:
installallthethingsononeserverconfig:
type: OS::Heat::SoftwareConfig
properties:
actions: [CREATE]
config: {get
file: installallthethingsononeserver.sh}

update_all_the_things_on_one_server_config:
  type: OS::Heat::SoftwareConfig
  properties:
    actions: [UPDATE]
    config: {get_file: update_all_the_things_on_one_server.sh}
    inputs:
      - name: update_count

...

(Filling in the rest is left as an exercise to the reader. You're welcome.)

I have to be honest, I'm not OpenStack Heat expert and the last block
is very complex to understand.
Maybe a simple example with Keystone API would have been great.

Not illustrated are the multiple sources of truth that we have: puppet
modules (packaged on the server), puppet manifests and hieradata (delivered
via Heat), external package repositories. Heat is a dataflow language but
much of the data it should be operating on is actually hidden from it.
That's going about as well as you might expect.

Due to the impossibility of ever rolling back a deployment like one of
those, we just disable rollback for the overcloud templates, so if there's a
failure we end up stuck in whatever intermediate state we were in when the
script died. That can leave things in an state where recovery is not
automatic when 'earlier' deployments (like the package update) end up
depending on state set up by 'later' deployments (like the post- scripts,
which manipulate Pacemaker's state in Pacemaker-based deployments). Even
worse, many of the current scripts leave the machine in a state that
requires manual recovery should they fail part-way through.

Indeed, this has literally none of the benefits of the ideal Heat deployment
enumerated above save one: it may be entirely the wrong tool in every way
for the job it's being asked to do, but at least it is still well-integrated
with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack', where we
have one stack for the infrastructure and a separate one for the software
deployments, so that there is no longer any tight integration between
infrastructure and software. Although it makes me a bit sad in some ways, I
can certainly appreciate the merits of the idea as well. However, from the
argument above we can deduce that if this is the only thing we do then we
will end up in the very worst of all possible worlds: the wrong tool for the
job, poorly integrated. Every single advantage of using Heat to deploy
software will have evaporated, leaving only disadvantages.

So what would be a good alternative? And how would we evaluate the options?

To my mind, the purpose of the TripleO project is this: to ensure that there
is an OpenStack community collaborating around each part of the OpenStack
installation/management story. We don't care about TripleO "owning" that
part (all things being equal, we'd prefer not to), just that nobody should
have to go outside the OpenStack community and/or roll their own thing to
install OpenStack unless they want to. So I think the ability to sustain a
community around whatever solution we choose ought to be a primary
consideration.

The use of Ironic has been something of a success story here. There's only
one place to add hardware support to enable both installing OpenStack itself
on bare-metal via TripleO and the 'regular' bare-metal-to-tenant use case of
Ironic. This is a clear win/win.

Beyond getting the bare-metal machines marshalled, the other part of the
solution is configuration management and orchestration of the various
software services. When TripleO started there was nowhere in OpenStack that
was defining the relationships between services needed to orchestrate them.
To a large extent there still isn't. I think that one of the reasons we
adopted Puppet in TripleO was that it was supposed to provide this, at least
within a limited scope (i.e. on one machine - the puppet-deploying community
is largely using Ansible to orchestrate across boxes, and we are using
Heat). However, what we've discovered in the past few months is that Puppet
is actually not able to fulfil this role as long as we support
Pacemaker-based deployments as an option, because in that case Pacemaker
actually has control of starting and stopping all of the services. As a
result we are back to defining it all ourselves in the Pacemaker config plus
various hacky shell scripts, instead of relying on (and contributing to!) a
larger community. Even ignoring that, Puppet doesn't solve the problem of
orchestrating across multiple machines.

That is True there was/is an overlap between Puppet & Pacemaker
orchestration for services management, that's why we disable service
management in Puppet.
This is something we are working on by created a Puppet provider to
control 'pcs'.

https://review.openstack.org/#/c/286124/

Thanks to this work, deployments done by Puppet will manage resources
using pcs CLI and we'll be able to cleanup a lot of hacks that made it
possible in the past.
I agree Pacemaker is not helping, it has nothing to do with the
deployment tool: Pacemaker is complex to deploy and manage.
On a positive way, we are working on improving the Pacemaker/Puppet
collision and I think we will fix that during Newton cycle.

Clearly one option would be to encode everything in Heat along the lines of
the first example above. I think once we have containers this could actually
work really well for compute nodes and other types of scale-out nodes (e.g.
Swift nodes). The scale-out model of Heat scaling groups works really well
for this use case, and between the improvements we have put in place (like
batched updates and user hooks) and those still on the agenda (like
notifications + automatic Mistral workflow triggering on hooks) Heat could
provide a really good way of capturing things like migrating user workloads
on scale down and rolling updates in the templates, so that they can be
managed completely automatically by the undercloud with no client
involvement (and when the undercloud becomes HA, they'll get HA for free).
I'd be pretty excited to see this tried. The potential downside is that the
orchestration definitions are still trapped inside the TripleO templates, so
they're not being shared outside of the TripleO community. This is probably
justified though owing to its close ties to the underlying infrastructure.

An alternative out of left field: as far as I can gather the "completely new
way of orchestrating activities" used by the new Puppet Application
Orchestration thing[2] uses substantially the same model as I described for
Heat above. If we added Puppet Application Orchestration data to
openstack-puppet-modules then it may be possible to write a tool to generate
Heat templates from that data. However in talking with Emilien it sounds
like o-p-m is quite some time away from tackling PAO. So I don't think this
is really feasible.

That is True, Puppet OpenStack modules do not support PAO and I
haven't seen any efforts from Puppetlabs to help us doing that.
I'm even not sure about how it would work, since it require a
Puppetmaster and some pieces are not Open-Source. Beurk.

In any event, it's when we get to the controller nodes that the downsides
become more pronounced. We're no longer talking about one deployment per
service like I sketched above; each service is actually multiple deployments
forming an active-active cluster with virtual IPs and failover and all that
jazz. It may be that everything would just work the same way, but we would
be in uncharted territory and there would likely be unanticipated
subtleties. It's particularly unclear how we would handle stop-the-world
database migrations in this model, although we do have the option of hoping
that stop-the-world database migrations will have been completely phased out
by then.

To make it even more complicated, we ultimately want the services to
heterogeneously spread among controller nodes in a configurable way. I
believe that Dan's work on composable roles has already gone some way toward
this without even using containers, but it's likely to become increasingly
difficult to model in Heat without some sort of template generation. (I
personally think that template generation would be a Good Thing, but we've
chosen not to go down that path so far.) Quite possibly even just having
composable roles could make it untenable to continue maintaining separate
Pacemaker and non-Pacemaker deployment modes. It'd be really nice to have
the flexibility to do things like scale out different services at different
rates. What's more, we are going to need some way of redistributing services
when a machine in the cluster fails, and ultimately we would like that
process to be automated, which would require a template generation
service.

We certainly could build all of that. But we definitely shouldn't because
this is the kind of thing that services like Kubernetes and Apache Mesos are
designed to do already. And that raises another possibility: Angus & friends
are working on capturing the orchestration relationships for Mesos+Marathon
within the Kolla project (specifically, in the kolla-mesos repository). This
represents a tremendous opportunity for the TripleO project to further its
mission of having the same deployment tools available to everyone as an
official part of the OpenStack project without having to maintain them
separately.

As of the Liberty release, Magnum now supports provisioning Mesos clusters,
so TripleO wouldn't have to maintain the installer for that either. (The
choice of Mesos is somewhat unfortunate in our case, because Magnum's
Kubernetes support is much more mature than its Mesos support, and because
the reasons for the decision are about to be or have already been overtaken
by events - I've heard reports that the features that Kubernetes was missing
to allow it to be used for controller nodes, and maybe even compute nodes,
are now available. Nonetheless, I expect the level of Magnum support for
Mesos is likely workable.) This is where the TripleO strategy of using
OpenStack to deploy OpenStack can really pay dividends: because we use
Ironic all of our servers are accessible through the Nova API, so in theory
we can just run Magnum out of the box.

This is something we already & partially discussed when I submitted
the Containers blueprint, in September 2015:
https://review.openstack.org/#/c/222002/

I was told to abandon this blueprint, so Container folks could create a new one:
https://review.openstack.org/#/c/223182/

The "new" blueprint was never merged so I guess TripleO team accepted
the design, moreoever discussed during Tokyo Summit.
I am sad to see you were not involved in both blueprints, nor present
during our discussions, but let's move forward.

The chances of me personally having time to prototype this are slim-to-zero,
but I think this is a path worth investigating.

A lot of people are currently very busy, because of releases and other
critical topics, like CI, split stack and composable roles, maybe
people focusing on containers can also reply to this thread and try to
understand why we're here now.

I know it has currently no impact on TripleO, but I plan to gather
feedback about Puppet & Containers usage during next Summit in Austin,
I know TimeWarner is deploying OpenStack with docker & Puppet
OpenStack modules at scale, and they have a pretty nice experience on
that.
--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

On Tue, Mar 22, 2016 at 6:01 PM, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

+1 for TripleO taking a look at Kolla.

Some random thoughts:

I'm in the middle of deploying a new cloud and I couldn't use either
TripleO or Kolla for various reasons. A few reasons for each:
* TripeO - worries me for ever having to do a major upgrade of the
software, or needing to do oddball configs like vxlans over ipoib.
* Kolla - At the time it was still immature. No stable artefacts posted.
database container recently broke, little documentation for disaster
recovery. No upgrade strategy at the time.

Kolla rearchitected recently to support oddball configs like we've had to
do at times. They also recently gained upgrade support. I think they are on
the right path. If I had to start fresh, I'd very seriously consider using
it.

I think Kolla can provide the missing pieces that TripleO needs. TripleO
has bare metal deployment down solid. I really like the idea of using
OpenStack to deploy OpenStack. Kolla is now OpenStack so should be
considered.

I'm also in favor of using Magnum to deploy a COE to manage Kolla. I'm
much less thrilled about Mesos though. It feels heavy enough weight that it
feels like your deploying an OpenStack like system just to deploy
OpenStack. So, OpenStack On NotOpenStack On OpenStack. :/ I've had good
luck with Kubernetes (much simpler) recently and am disappointed that it
was too immature at the time Kolla originally considered it. It seems much
more feasible to use now. I use net=host like features all the time which
was a major sticking point before.

Frankly speaking, I think that Kolla project is doing right thing keeping
its component decoupled. There are two parts of Kolla-mesos approach: use
micro-services approach inside containers (service discovery,
self-configuration etc..) and use Mesos/Marathon definitions to control
containers placement. If you want to use Kubernetes you still can use Kolla
images and you will need to write simple PODs\ReplicationGroups
definitions. OpenStack services will configure themselves by using central
configuration storage (ZooKeeper for now but other storages are on their
way in oslo.config BP
https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db )

I'd be interested in seeing TripeO use the Ansible version for now since
that's working, stable, and supports upgrades/oddball configs. Then in the
future as Kubernetes support or maybe Mesos support matures, consider that.
Kolla's going to have to have a migration path from one to the other
eventually... I think this would allow TripeO to really come into its own
as an end to end, production ready system sooner.

Thanks,
Kevin


From: Emilien Macchi [emilien@redhat.com]
Sent: Tuesday, March 22, 2016 5:18 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of
Heat, containers, and the future of TripleO

This is quite a long mail, I'll reply to some statements inline.

On Mon, Mar 21, 2016 at 4:14 PM, Zane Bitter zbitter@redhat.com wrote:

tl;dr Containers represent a massive, and also mandatory, opportunity for
TripleO. Lets start thinking about ways that we can take maximum
advantage
to achieve the goals of the project.

Now that you have the tl;dr I'm going to start from the beginning, so
settle
in and grab yourself a cup of coffee or other poison of your choice.

After working on developing Heat from the very beginning of the project
in
early 2012 and debugging a bunch of TripleO deployments in the field, it
is
my considered opinion that Heat is a poor fit for the workloads that
TripleO
is currently asking of it. To illustrate why, I need to explain what it
is
that Heat is really designed to do.

Here's a theoretical example of how I've always imagined Heat software
deployments would make Heat users' lives better. For simplicity, I'm just
going to model two software components, a user-facing service that
connects
to some back-end service:

resources:
backendcomponent:
type: OS::Heat::SoftwareComponent
properties:
configs:
- tool: script
actions:
- CREATE
- UPDATE
config: |
PORT=$(get
backendport || randomport)
stopbackend
start
backend $DEPLOYVERSION $PORT $CONFIG
addr="$(hostname):$(get
backendport)"
printf '%s' "$addr" >${heat
outputspath}.hostandport
- tool: script
actions:
- DELETE
config: |
stop
backend
inputs:
- name: DEPLOYVERSION
- name: CONFIG
outputs:
- name: host
and_port

frontend_component:
  type: OS::Heat::SoftwareComponent
  properties:
    configs:
      - tool: script
        actions:
          - CREATE
          - UPDATE
        config: |
          stop_frontend
          start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG
      - tool: script
        actions:
          - DELETE
        config: |
          stop_frontend
    inputs:
      - name: DEPLOY_VERSION
      - name: BACKEND_ADDR
      - name: CONFIG

backend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: backend_server}
    name: {get_param: backend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: backend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: backend_version}
      CONFIG: ${get_param: backend_config}

frontend:
  type: OS::Heat::SoftwareDeployment
  properties:
    server: {get_resource: frontend_server}
    name: {get_param: frontend_version} # Forces upgrade replacement
    actions: [CREATE, UPDATE, DELETE]
    config: {get_resource: frontend_component}
    input_values:
      DEPLOY_VERSION: ${get_param: frontend_version}
      BACKEND_ADDR: {get_attr: [backend, host_and_port]}
      CONFIG: ${get_param: frontend_config}

This is actually quite a beautiful system, if I may say so:

  • Whenever a version changes, Heat knows to update that component, and
    the
    components can be updated independently.
  • If the backend in this example restarts on a different port, the
    frontend
    is updated to point to the new port.
  • Everything is completely agnostic as to which server it is running on.
    They could be running on the same server or different servers.
  • Everything is integrated with the infrastructure (not only the servers
    you're deploying on and the networks and volumes connected to them, but
    also
    things like load balancers), so everything is created at the right time,
    in
    parallel where possible, and any errors are reported all in one place.
  • If something requires e.g. a restart after changing another component,
    we
    can encode that. And if it doesn't, we can encode that too.
  • There's next to no downtime required: if e.g. we upgrade the backend,
    we
    first deploy a new one listening on a new port, then update the frontend
    to
    listen on the new port, then finally shut down the old backend. Again, we
    can choose when we want this and when we just want to update in place and
    reload.
  • The application doesn't even need to worry about versioning the
    protocol
    that its two constituent parts communicate over: as long as the
    backendversion and frontendversion that we pass are always compatible,
    only compatible versions of the two services ever talk to each other.
  • If anything at all fails at any point before, during or after this
    part of
    the template, Heat can automatically roll everything back into the exact
    same state as it was in before, without any outside intervention. You can
    insert test deployments that check everything is working and have them
    automatically roll back if it's not, all with no downtime for users.

So you can use this to do something like a fancier version of blue-green
deployment,[1] where you're actually rolling out the (virtualised)
hardware
and infrastructure in a blue-green fashion along with the software. Not
only
that, you can choose to replace your whole stack or only parts of it.
(Note:
the way I had to encode this in the example above, by changing the
deployment name so that it forces a resource replacement, is a hack. We
really need a feature to specify in a software config resource which
inputs
should result in a replacement on change.)

It's worth noting that in practice you really, really want everything
deployed in containers to make this process work consistently, even
though
in theory you could make this work (briefly) without them. In
particular,
rollback without containers is a dicey proposition. When we first started
talking about implementing software deployments in Heat I half-seriously
suggested that maybe we should make containers the only allowed type of
software deployment, and I kind of wonder now if I shouldn't have pressed
harder on that point.

In any event, unfortunately as everyone involved in TripleO knows, the
way
TripleO uses Heat looks nothing like this. It actually looks more like
this:

resources:
installallthethingsononeserverconfig:
type: OS::Heat::SoftwareConfig
properties:
actions: [CREATE]
config: {get
file: installallthethingsononeserver.sh}

update_all_the_things_on_one_server_config:
  type: OS::Heat::SoftwareConfig
  properties:
    actions: [UPDATE]
    config: {get_file: update_all_the_things_on_one_server.sh}
    inputs:
      - name: update_count

...

(Filling in the rest is left as an exercise to the reader. You're
welcome.)

I have to be honest, I'm not OpenStack Heat expert and the last block
is very complex to understand.
Maybe a simple example with Keystone API would have been great.

Not illustrated are the multiple sources of truth that we have: puppet
modules (packaged on the server), puppet manifests and hieradata
(delivered
via Heat), external package repositories. Heat is a dataflow language but
much of the data it should be operating on is actually hidden from it.
That's going about as well as you might expect.

Due to the impossibility of ever rolling back a deployment like one of
those, we just disable rollback for the overcloud templates, so if
there's a
failure we end up stuck in whatever intermediate state we were in when
the
script died. That can leave things in an state where recovery is not
automatic when 'earlier' deployments (like the package update) end up
depending on state set up by 'later' deployments (like the post- scripts,
which manipulate Pacemaker's state in Pacemaker-based deployments). Even
worse, many of the current scripts leave the machine in a state that
requires manual recovery should they fail part-way through.

Indeed, this has literally none of the benefits of the ideal Heat
deployment
enumerated above save one: it may be entirely the wrong tool in every way
for the job it's being asked to do, but at least it is still
well-integrated
with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we
have one stack for the infrastructure and a separate one for the software
deployments, so that there is no longer any tight integration between
infrastructure and software. Although it makes me a bit sad in some
ways, I
can certainly appreciate the merits of the idea as well. However, from
the
argument above we can deduce that if this is the only thing we do then
we
will end up in the very worst of all possible worlds: the wrong tool for
the
job, poorly integrated. Every single advantage of using Heat to deploy
software will have evaporated, leaving only disadvantages.

So what would be a good alternative? And how would we evaluate the
options?

To my mind, the purpose of the TripleO project is this: to ensure that
there
is an OpenStack community collaborating around each part of the OpenStack
installation/management story. We don't care about TripleO "owning" that
part (all things being equal, we'd prefer not to), just that nobody
should
have to go outside the OpenStack community and/or roll their own thing to
install OpenStack unless they want to. So I think the ability to sustain
a
community around whatever solution we choose ought to be a primary
consideration.

The use of Ironic has been something of a success story here. There's
only
one place to add hardware support to enable both installing OpenStack
itself
on bare-metal via TripleO and the 'regular' bare-metal-to-tenant use
case of
Ironic. This is a clear win/win.

Beyond getting the bare-metal machines marshalled, the other part of the
solution is configuration management and orchestration of the various
software services. When TripleO started there was nowhere in OpenStack
that
was defining the relationships between services needed to orchestrate
them.
To a large extent there still isn't. I think that one of the reasons we
adopted Puppet in TripleO was that it was supposed to provide this, at
least
within a limited scope (i.e. on one machine - the puppet-deploying
community
is largely using Ansible to orchestrate across boxes, and we are using
Heat). However, what we've discovered in the past few months is that
Puppet
is actually not able to fulfil this role as long as we support
Pacemaker-based deployments as an option, because in that case Pacemaker
actually has control of starting and stopping all of the services. As a
result we are back to defining it all ourselves in the Pacemaker config
plus
various hacky shell scripts, instead of relying on (and contributing
to!) a
larger community. Even ignoring that, Puppet doesn't solve the problem of
orchestrating across multiple machines.

That is True there was/is an overlap between Puppet & Pacemaker
orchestration for services management, that's why we disable service
management in Puppet.
This is something we are working on by created a Puppet provider to
control 'pcs'.

https://review.openstack.org/#/c/286124/

Thanks to this work, deployments done by Puppet will manage resources
using pcs CLI and we'll be able to cleanup a lot of hacks that made it
possible in the past.
I agree Pacemaker is not helping, it has nothing to do with the
deployment tool: Pacemaker is complex to deploy and manage.
On a positive way, we are working on improving the Pacemaker/Puppet
collision and I think we will fix that during Newton cycle.

Clearly one option would be to encode everything in Heat along the lines
of
the first example above. I think once we have containers this could
actually
work really well for compute nodes and other types of scale-out nodes
(e.g.
Swift nodes). The scale-out model of Heat scaling groups works really
well
for this use case, and between the improvements we have put in place
(like
batched updates and user hooks) and those still on the agenda (like
notifications + automatic Mistral workflow triggering on hooks) Heat
could
provide a really good way of capturing things like migrating user
workloads
on scale down and rolling updates in the templates, so that they can be
managed completely automatically by the undercloud with no client
involvement (and when the undercloud becomes HA, they'll get HA for
free).
I'd be pretty excited to see this tried. The potential downside is that
the
orchestration definitions are still trapped inside the TripleO
templates, so
they're not being shared outside of the TripleO community. This is
probably
justified though owing to its close ties to the underlying
infrastructure.

An alternative out of left field: as far as I can gather the "completely
new
way of orchestrating activities" used by the new Puppet Application
Orchestration thing[2] uses substantially the same model as I described
for
Heat above. If we added Puppet Application Orchestration data to
openstack-puppet-modules then it may be possible to write a tool to
generate
Heat templates from that data. However in talking with Emilien it sounds
like o-p-m is quite some time away from tackling PAO. So I don't think
this
is really feasible.

That is True, Puppet OpenStack modules do not support PAO and I
haven't seen any efforts from Puppetlabs to help us doing that.
I'm even not sure about how it would work, since it require a
Puppetmaster and some pieces are not Open-Source. Beurk.

In any event, it's when we get to the controller nodes that the downsides
become more pronounced. We're no longer talking about one deployment per
service like I sketched above; each service is actually multiple
deployments
forming an active-active cluster with virtual IPs and failover and all
that
jazz. It may be that everything would just work the same way, but we
would
be in uncharted territory and there would likely be unanticipated
subtleties. It's particularly unclear how we would handle stop-the-world
database migrations in this model, although we do have the option of
hoping
that stop-the-world database migrations will have been completely phased
out
by then.

To make it even more complicated, we ultimately want the services to
heterogeneously spread among controller nodes in a configurable way. I
believe that Dan's work on composable roles has already gone some way
toward
this without even using containers, but it's likely to become
increasingly
difficult to model in Heat without some sort of template generation. (I
personally think that template generation would be a Good Thing, but
we've
chosen not to go down that path so far.) Quite possibly even just having
composable roles could make it untenable to continue maintaining separate
Pacemaker and non-Pacemaker deployment modes. It'd be really nice to have
the flexibility to do things like scale out different services at
different
rates. What's more, we are going to need some way of redistributing
services
when a machine in the cluster fails, and ultimately we would like that
process to be automated, which would require a template generation
service.

We certainly could build all of that. But we definitely shouldn't
because
this is the kind of thing that services like Kubernetes and Apache Mesos
are
designed to do already. And that raises another possibility: Angus &
friends
are working on capturing the orchestration relationships for
Mesos+Marathon
within the Kolla project (specifically, in the kolla-mesos repository).
This
represents a tremendous opportunity for the TripleO project to further
its
mission of having the same deployment tools available to everyone as an
official part of the OpenStack project without having to maintain them
separately.

As of the Liberty release, Magnum now supports provisioning Mesos
clusters,
so TripleO wouldn't have to maintain the installer for that either. (The
choice of Mesos is somewhat unfortunate in our case, because Magnum's
Kubernetes support is much more mature than its Mesos support, and
because
the reasons for the decision are about to be or have already been
overtaken
by events - I've heard reports that the features that Kubernetes was
missing
to allow it to be used for controller nodes, and maybe even compute
nodes,
are now available. Nonetheless, I expect the level of Magnum support for
Mesos is likely workable.) This is where the TripleO strategy of using
OpenStack to deploy OpenStack can really pay dividends: because we use
Ironic all of our servers are accessible through the Nova API, so in
theory
we can just run Magnum out of the box.

This is something we already & partially discussed when I submitted
the Containers blueprint, in September 2015:
https://review.openstack.org/#/c/222002/

I was told to abandon this blueprint, so Container folks could create a
new one:
https://review.openstack.org/#/c/223182/

The "new" blueprint was never merged so I guess TripleO team accepted
the design, moreoever discussed during Tokyo Summit.
I am sad to see you were not involved in both blueprints, nor present
during our discussions, but let's move forward.

The chances of me personally having time to prototype this are
slim-to-zero,
but I think this is a path worth investigating.

A lot of people are currently very busy, because of releases and other
critical topics, like CI, split stack and composable roles, maybe
people focusing on containers can also reply to this thread and try to
understand why we're here now.

I know it has currently no impact on TripleO, but I plan to gather
feedback about Puppet & Containers usage during next Summit in Austin,
I know TimeWarner is deploying OpenStack with docker & Puppet
OpenStack modules at scale, and they have a pretty nice experience on
that.
--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Georgy Okrokvertskhov
Director of Performance Engineering,
OpenStack Platform Products,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Georgy_Okrokvertskho (3,820 points)   2 4
0 votes

Snip

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.

I think Heat is a very powerful tool having done the container integration
into the tripleo-heat-templates I can see its appeal. Something I learned
from integration, was that Heat is not the best tool for container deployment,
at least right now. We were able to leverage the work in Kolla, but what it
came down to was that we're not using containers or Kolla to its max potential.

I did an evaluation recently of tripleo and kolla to see what we would gain
if the two were to combine. Let's look at some items on tripleo's roadmap.
Split stack, as mentioned above, would be gained if tripleo were to adopt
Kolla. Tripleo holds the undercloud and ironic. Kolla separates config
and deployment. Therefore, allowing for the decoupling for each piece of
the stack. Composable roles, this would be the ability to land services
onto separate hosts on demand. Kolla also already does this [1]. Finally,
container integration, this is just a given :).

In the near term, if tripleo were to adopt Kolla as its overcloud it would
be provided these features and retire heat to setting up the baremetal nodes
and providing those ips to ansible. This would be great for kolla too because
it would provide baremetal provisioning.

Ian Main and I are currently working on a POC for this as of last week [2].
It's just a simple heat template :).

I think further down the road we can evaluate using kubernetes [3].
For now though, kolla-anisble is rock solid and is worth using for the
overcloud.

Thanks!
-Ryan

[1] - https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode
[2] - https://github.com/rthallisey/kolla-heat-templates
[3] - https://review.openstack.org/#/c/255450/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Ryan_Hallisey (2,980 points)   1 3
0 votes

Hello,

So Ryan, I think you can make use of heat all the way. Architecture of
kolla doesn't require you to use ansible at all (in fact, we separate
ansible code to a different repo). Truth is that ansible-kolla is
developed by most people and considered "the way to deploy kolla" by
most of us, but we make sure that we won't cut out other deployment
engines from our potential.

So bottom line, heat may very well replace ansible code if you can
duplicate logic we have in playbooks in heat templates. That may
require docker resource with pretty complete featureset of docker
itself (named volumes being most important). Bootstrap is usually done
inside container, so that would be possible too.

To be honest, as for tripleo doing just bare metal deployment would
defeat idea of tripleo. We have bare metal deployment tools already
(cobbler which is used widely, bifrost which use ansible same as kolla
and integration would be easier), and these comes with significantly
less footprint than whole tripleo infrastructure. Strength of tripleo
comes from it's rich config of openstack itself, and I think that
should be portable to kolla.

On 23 March 2016 at 06:54, Ryan Hallisey rhallise@redhat.com wrote:
Snip

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.

I think Heat is a very powerful tool having done the container integration
into the tripleo-heat-templates I can see its appeal. Something I learned
from integration, was that Heat is not the best tool for container deployment,
at least right now. We were able to leverage the work in Kolla, but what it
came down to was that we're not using containers or Kolla to its max potential.

I did an evaluation recently of tripleo and kolla to see what we would gain
if the two were to combine. Let's look at some items on tripleo's roadmap.
Split stack, as mentioned above, would be gained if tripleo were to adopt
Kolla. Tripleo holds the undercloud and ironic. Kolla separates config
and deployment. Therefore, allowing for the decoupling for each piece of
the stack. Composable roles, this would be the ability to land services
onto separate hosts on demand. Kolla also already does this [1]. Finally,
container integration, this is just a given :).

In the near term, if tripleo were to adopt Kolla as its overcloud it would
be provided these features and retire heat to setting up the baremetal nodes
and providing those ips to ansible. This would be great for kolla too because
it would provide baremetal provisioning.

Ian Main and I are currently working on a POC for this as of last week [2].
It's just a simple heat template :).

I think further down the road we can evaluate using kubernetes [3].
For now though, kolla-anisble is rock solid and is worth using for the
overcloud.

Thanks!
-Ryan

[1] - https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode
[2] - https://github.com/rthallisey/kolla-heat-templates
[3] - https://review.openstack.org/#/c/255450/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Michał_Jastrzębski (9,220 points)   1 5 5
0 votes

On 03/23/2016 11:42 AM, Michał Jastrzębski wrote:
Hello,

So Ryan, I think you can make use of heat all the way. Architecture of
kolla doesn't require you to use ansible at all (in fact, we separate
ansible code to a different repo). Truth is that ansible-kolla is
developed by most people and considered "the way to deploy kolla" by
most of us, but we make sure that we won't cut out other deployment
engines from our potential.

So bottom line, heat may very well replace ansible code if you can
duplicate logic we have in playbooks in heat templates. That may
require docker resource with pretty complete featureset of docker
itself (named volumes being most important). Bootstrap is usually done
inside container, so that would be possible too.

Heat can call Anisble.

Why would it not be Heats responsibility for creating the stack, and
then Kolla-ansible for setting everything up?

Heat is more esoteric than Ansible. I expcet the amount of people that
know and use Ansible to far outweight the number of people that know
Heat. Let's make it easy for them to get involved. Use each as
appropriate, but let the config with Heat clearly map to a config
without it for a Kolla based deploy.

To be honest, as for tripleo doing just bare metal deployment would
defeat idea of tripleo. We have bare metal deployment tools already
(cobbler which is used widely, bifrost which use ansible same as kolla
and integration would be easier), and these comes with significantly
less footprint than whole tripleo infrastructure. Strength of tripleo
comes from it's rich config of openstack itself, and I think that
should be portable to kolla.

On 23 March 2016 at 06:54, Ryan Hallisey rhallise@redhat.com wrote:

Snip

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.
Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.
I think Heat is a very powerful tool having done the container integration
into the tripleo-heat-templates I can see its appeal. Something I learned
from integration, was that Heat is not the best tool for container deployment,
at least right now. We were able to leverage the work in Kolla, but what it
came down to was that we're not using containers or Kolla to its max potential.

I did an evaluation recently of tripleo and kolla to see what we would gain
if the two were to combine. Let's look at some items on tripleo's roadmap.
Split stack, as mentioned above, would be gained if tripleo were to adopt
Kolla. Tripleo holds the undercloud and ironic. Kolla separates config
and deployment. Therefore, allowing for the decoupling for each piece of
the stack. Composable roles, this would be the ability to land services
onto separate hosts on demand. Kolla also already does this [1]. Finally,
container integration, this is just a given :).

In the near term, if tripleo were to adopt Kolla as its overcloud it would
be provided these features and retire heat to setting up the baremetal nodes
and providing those ips to ansible. This would be great for kolla too because
it would provide baremetal provisioning.

Ian Main and I are currently working on a POC for this as of last week [2].
It's just a simple heat template :).

I think further down the road we can evaluate using kubernetes [3].
For now though, kolla-anisble is rock solid and is worth using for the
overcloud.

Thanks!
-Ryan

[1] - https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode
[2] - https://github.com/rthallisey/kolla-heat-templates
[3] - https://review.openstack.org/#/c/255450/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Adam_Young (19,940 points)   2 7 9
0 votes

Fox, Kevin M wrote:
+1 for TripleO taking a look at Kolla.

Some random thoughts:

I'm in the middle of deploying a new cloud and I couldn't use either TripleO or Kolla for various reasons. A few reasons for each:
* TripeO - worries me for ever having to do a major upgrade of the software, or needing to do oddball configs like vxlans over ipoib.
* Kolla - At the time it was still immature. No stable artefacts posted. database container recently broke, little documentation for disaster recovery. No upgrade strategy at the time.

Kolla rearchitected recently to support oddball configs like we've had to do at times. They also recently gained upgrade support. I think they are on the right path. If I had to start fresh, I'd very seriously consider using it.

I think Kolla can provide the missing pieces that TripleO needs. TripleO has bare metal deployment down solid. I really like the idea of using OpenStack to deploy OpenStack. Kolla is now OpenStack so should be considered.

I'm also in favor of using Magnum to deploy a COE to manage Kolla. I'm much less thrilled about Mesos though. It feels heavy enough weight that it feels like your deploying an OpenStack like system just to deploy OpenStack. So, OpenStack On NotOpenStack On OpenStack. :/ I've had good luck with Kubernetes (much simpler) recently and am disappointed that it was too immature at the time Kolla originally considered it. It seems much more feasible to use now. I use net=host like features all the time which was a major sticking point before.

I'd be interested in seeing TripeO use the Ansible version for now since that's working, stable, and supports upgrades/oddball configs. Then in the future as Kubernetes support or maybe Mesos support matures, consider that. Kolla's going to have to have a migration path from one to the other eventually... I think this would allow TripeO to really come into its own as an end to end, production ready system sooner.

Thanks,
Kevin

This is very much my thinking as well. I like your pragmatic take on it.
The community is building solutions to these problems and we should join
them.

Ian


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Ian_Main (260 points)   1
0 votes

On 23/03/16 07:54, Ryan Hallisey wrote:
Snip

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.

I think Heat is a very powerful tool having done the container integration
into the tripleo-heat-templates I can see its appeal. Something I learned
from integration, was that Heat is not the best tool for container deployment,
at least right now. We were able to leverage the work in Kolla, but what it
came down to was that we're not using containers or Kolla to its max potential.

I did an evaluation recently of tripleo and kolla to see what we would gain
if the two were to combine. Let's look at some items on tripleo's roadmap.
Split stack, as mentioned above, would be gained if tripleo were to adopt
Kolla. Tripleo holds the undercloud and ironic. Kolla separates config
and deployment. Therefore, allowing for the decoupling for each piece of
the stack. Composable roles, this would be the ability to land services
onto separate hosts on demand. Kolla also already does this [1]. Finally,
container integration, this is just a given :).

In the near term, if tripleo were to adopt Kolla as its overcloud it would
be provided these features and retire heat to setting up the baremetal nodes
and providing those ips to ansible. This would be great for kolla too because
it would provide baremetal provisioning.

Ian Main and I are currently working on a POC for this as of last week [2].
It's just a simple heat template :).

I think further down the road we can evaluate using kubernetes [3].
For now though, kolla-anisble is rock solid and is worth using for the
overcloud.

My concern about kolla-ansible is that the requirements might start
getting away from what the original design was intended to cope with,
and that it may prove difficult to extend. For example, I wrote about
the idea of doing the container deployments with pure Heat:

What's more, we are going to need some way of redistributing services when a machine in the cluster fails, and ultimately we would like that process to be automated, which would require a template generation service.

We certainly could build all of that. But we definitely shouldn't

and to my mind kolla-ansible is in a similar category in that respect
(it does, of course, have an existing community and in that sense is
still strictly superior to the pure-Heat approach). There's lots of
stuff in e.g. Kubernetes that it seems likely we'll want and, while
there's no theoretical obstacle to implementing them in Ansible, these
are hard, subtle problems which are presumably better left to a
specialist project.

I'd be happy to hear other opinions on that though. Maybe we don't care
about any of that container cluster management stuff, and if something
fails we just let everything run degraded until we can pull in a
replacement? I don't know.

cheers,
Zane.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Zane_Bitter (21,640 points)   4 6 9
0 votes

If heat convergence worked (Is that a thing yet?), it could potentially be used instead of a COE like kubernetes.

The thing ansible buys us today would be upgradeability. Ansible is config management, but its also a workflow like tool. Heats bad at workflow.

I think between Heat with Convergence, Kolla containers, and some kind of Mistral workflow for upgrades, you could replace Ansible.

Then there's the nova instance user thing again (https://review.openstack.org/222293)... How do you get secrets to the instances securely... Kubernetes has a secure store we could use... OpenStack still hasn't really gotten this one figured out. :/ Barbican is a piece of that puzzle, but there's no really good to hook it and nova together.

Thanks,
Kevin


From: Michał Jastrzębski [inc007@gmail.com]
Sent: Wednesday, March 23, 2016 8:42 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of Heat, containers, and the future of TripleO

Hello,

So Ryan, I think you can make use of heat all the way. Architecture of
kolla doesn't require you to use ansible at all (in fact, we separate
ansible code to a different repo). Truth is that ansible-kolla is
developed by most people and considered "the way to deploy kolla" by
most of us, but we make sure that we won't cut out other deployment
engines from our potential.

So bottom line, heat may very well replace ansible code if you can
duplicate logic we have in playbooks in heat templates. That may
require docker resource with pretty complete featureset of docker
itself (named volumes being most important). Bootstrap is usually done
inside container, so that would be possible too.

To be honest, as for tripleo doing just bare metal deployment would
defeat idea of tripleo. We have bare metal deployment tools already
(cobbler which is used widely, bifrost which use ansible same as kolla
and integration would be easier), and these comes with significantly
less footprint than whole tripleo infrastructure. Strength of tripleo
comes from it's rich config of openstack itself, and I think that
should be portable to kolla.

On 23 March 2016 at 06:54, Ryan Hallisey rhallise@redhat.com wrote:
Snip

Indeed, this has literally none of the benefits of the ideal Heat
deployment enumerated above save one: it may be entirely the wrong tool
in every way for the job it's being asked to do, but at least it is
still well-integrated with the rest of the infrastructure.

Now, at the Mitaka summit we discussed the idea of a 'split stack',
where we have one stack for the infrastructure and a separate one for
the software deployments, so that there is no longer any tight
integration between infrastructure and software. Although it makes me a
bit sad in some ways, I can certainly appreciate the merits of the idea
as well. However, from the argument above we can deduce that if this is
the only thing we do then we will end up in the very worst of all
possible worlds: the wrong tool for the job, poorly integrated. Every
single advantage of using Heat to deploy software will have evaporated,
leaving only disadvantages.

I think Heat is a very powerful tool having done the container integration
into the tripleo-heat-templates I can see its appeal. Something I learned
from integration, was that Heat is not the best tool for container deployment,
at least right now. We were able to leverage the work in Kolla, but what it
came down to was that we're not using containers or Kolla to its max potential.

I did an evaluation recently of tripleo and kolla to see what we would gain
if the two were to combine. Let's look at some items on tripleo's roadmap.
Split stack, as mentioned above, would be gained if tripleo were to adopt
Kolla. Tripleo holds the undercloud and ironic. Kolla separates config
and deployment. Therefore, allowing for the decoupling for each piece of
the stack. Composable roles, this would be the ability to land services
onto separate hosts on demand. Kolla also already does this [1]. Finally,
container integration, this is just a given :).

In the near term, if tripleo were to adopt Kolla as its overcloud it would
be provided these features and retire heat to setting up the baremetal nodes
and providing those ips to ansible. This would be great for kolla too because
it would provide baremetal provisioning.

Ian Main and I are currently working on a POC for this as of last week [2].
It's just a simple heat template :).

I think further down the road we can evaluate using kubernetes [3].
For now though, kolla-anisble is rock solid and is worth using for the
overcloud.

Thanks!
-Ryan

[1] - https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode
[2] - https://github.com/rthallisey/kolla-heat-templates
[3] - https://review.openstack.org/#/c/255450/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Hello,

So Ryan, I think you can make use of heat all the way. Architecture of
kolla doesn't require you to use ansible at all (in fact, we separate
ansible code to a different repo). Truth is that ansible-kolla is
developed by most people and considered "the way to deploy kolla" by
most of us, but we make sure that we won't cut out other deployment
engines from our potential.

So bottom line, heat may very well replace ansible code if you can
duplicate logic we have in playbooks in heat templates. That may
require docker resource with pretty complete featureset of docker
itself (named volumes being most important). Bootstrap is usually done
inside container, so that would be possible too.

Heat can call Anisble.

Why would it not be Heats responsibility for creating the stack, and
then Kolla-ansible for setting everything up?

Heat is more esoteric than Ansible. I expcet the amount of people that
know and use Ansible to far outweight the number of people that know
Heat. Let's make it easy for them to get involved. Use each as
appropriate, but let the config with Heat clearly map to a config
without it for a Kolla based deploy.

I didn't know heat can call Ansible. Now that I know that let me refine.
I think it would be nice to have heat use kolla-ansible.

With split-stack/composable-roles, the tripleo-heat-templates are going
to undergo major reconstruction. So then the questions are, do we
construct the templates to 1) use kolla-ansible or 2) rewrite them with
kolla-ansible logic in heat or 3) go with kolla-kubernetes.

1) This solution involves merging the kolla and tripleo communities.
kolla-tripleo maybe? This path will come to a solution significantly faster
since it will be completely leveraging the work kolla has done. I think
ansible is a good tool, but I don't know if it's the best for container
deployment/management.

2) This solution is right along the lines of dprince's patch series [1],
but with focus on deploying kolla's containers. This option has a longer
road than 1. I think it could work and I think it would be a good
solution.

I'd be happy to hear other opinions on that though. Maybe we don't care
about any of that container cluster management stuff, and if something
fails we just let everything run degraded until we can pull in a
replacement? I don't know.

3) Kolla-kubernetes is only a spec at this point, but with kubernetes the
undercloud would use magnum. This option to me, has the most upside, but
the longest road. I think the cluster management tools: replication
controllers, health checks, deployments, ect., would be great additions.

My excitement around kolla-ansible stems from the fact that it is significantly
farther along than kolla-kubernetes. I haven't done a deployment of
kolla-kuberenetes since we dropped it a year ago. But having done an evaluation
of it recently, I think it's the best option long term. Until I use it with
kolla + magnum, I won't know for certain.

Thanks,
-Ryan

[1] - https://review.openstack.org/#/c/295588/5


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 23, 2016 by Ryan_Hallisey (2,980 points)   1 3
...