settingsLogin | Registersettings

[openstack-dev] [TripleO] a new Undercloud install driven by Heat

0 votes

Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.

I created a short video demonstration which goes over some of the
history behind the approach, and shows a live demo of all of this
working with the patches above:

https://www.youtube.com/watch?v=y1qMDLAf26Q

Thoughts? Would it be cool to have a session to discuss this more in
Barcelona?

Dan Prince (dprince)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 4, 2016 in openstack-dev by Dan_Prince (8,160 points)   1 5 8
retagged Jan 26, 2017 by admin

16 Responses

0 votes

On 08/04/2016 11:48 PM, Dan Prince wrote:
Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.

I don't want to sound rude, but please no. The fact that you have a
hammer does not mean everything around is nails :( What problem are you
trying to solve by doing it?

Undercloud installation is already sometimes fragile, but it's probably
the least fragile part right now (at least from my experience) And at
the very least it's pretty obviously debuggable in most cases. THT is
hard to understand and often impossible to debug. I'd prefer we move
away from THT completely rather than trying to fix it in one more place
where heat does not fit..

I created a short video demonstration which goes over some of the
history behind the approach, and shows a live demo of all of this
working with the patches above:

https://www.youtube.com/watch?v=y1qMDLAf26Q

Thoughts? Would it be cool to have a session to discuss this more in
Barcelona?

Dan Prince (dprince)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dmitry_Tantsur (18,080 points)   2 4 8
0 votes

On Fri, Aug 05, 2016 at 12:27:40PM +0200, Dmitry Tantsur wrote:
On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.

I don't want to sound rude, but please no. The fact that you have a hammer
does not mean everything around is nails :( What problem are you trying to
solve by doing it?

I think Dan explains it pretty well in his video, and your comment
indicates a fundamental misunderstanding around the entire TripleO vision,
which is about symmetry and reuse between deployment tooling and the
deployed cloud.

The problems this would solve are several:

  1. Remove divergence between undercloud and overcloud puppet implementation
    (instead of having an undercloud specific manifest, we reuse the exact
    same stuff we use for overcloud deployments)

  2. Better modularity, far easier to enable/disable services

  3. Get container integration "for free" when we land it in the overcloud

  4. Any introspection and debugging workflow becomes identical between the
    undercloud and overcloud

  5. We remove dependencies on a bunch of legacy scripts which run outside of
    puppet

  6. Whenever someone lands support for a new service in the overcloud, we
    automatically get undercloud support for it, completely for free.

  7. Potential for much easier implementation of a multi-node undercloud

Undercloud installation is already sometimes fragile, but it's probably the
least fragile part right now (at least from my experience) And at the very
least it's pretty obviously debuggable in most cases. THT is hard to
understand and often impossible to debug. I'd prefer we move away from THT
completely rather than trying to fix it in one more place where heat does
not fit..

These are some strong but unqualified assertions, so it's hard to really
respond. Yes, there is complexity, but it's a set of abstractions which
actually work pretty well for us, so there is value in having just one set
of abstractions used everywhere vs special-casing the undercloud.

Re moving away from THT completely, this is not a useful statement -
yes, there are alternative tools, but if you were to remove THT and just
use some other tool with Ironic, the result would simply not be TripleO.
There would be zero migration/upgrade path for existing users and all
third-party integrations (and our API/UI) would break.

FWIW I think this prototyping work is very interesting, and I'm certainly
keen to get wider (more constructive) feedback and see where it leads.

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Steven_Hardy (16,900 points)   2 7 12
0 votes

On Fri, 2016-08-05 at 12:27 +0200, Dmitry Tantsur wrote:
On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new way
to
install the Undercloud. The driving force behind this was some of
the
recent "composable services" work we've done in TripleO so
initially I
called in composable undercloud. There is an etherpad here with
links
to some of the patches already posted upstream (many of which stand
as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process
all-
in-one heat-all (engine and API) and thereby avoid things like
Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.
I don't want to sound rude, but please no. The fact that you have a 
hammer does not mean everything around is nails :( What problem are
you 
trying to solve by doing it?

Several problems I think.

One is TripleO has gradually moved away from elements. And while we
still use DIB elements for some things we no longer favor that tool and
instead rely on Heat and config management tooling to do our stepwise
deployment ordering. This leaves us using instack-undercloud a tool
built specifically to install elements locally as a means to create our
undercloud. It works... and I do think we've packaged it nicely but it
isn't the best architectural fit for where we are going I think. I
actually think that from an end/user contribution standpoint using t-h-
t could be quite nice for adding features to the Undercloud.

Second would be re-use. We just spent a huge amount of time in Newton
(and some in Mitaka) refactoring t-h-t around composable services. So
say you add a new composable service for Barbican in the Overcloud...
wouldn't it be nice to be able to consume the same thing in your
Undercloud as well? Right now you can't, you have to do some of the
work twice and in quite different formats I think. Sure, there is some
amount of shared puppet work but that is only part of the picture I
think.

There are new features to think about here too. Once upon a time
TripleO supported multi-node underclouds. When we switched to instack-
undercloud we moved away from that. By switching back to tripleo-heat-
templates we could structure our templates around abstractions like
resource groups and the new 'deployed-server' trick that allow you to
create machines either locally or perhaps via Ironic too. We could
avoid Ironic entirely and always install the Undercloud on existing
servers via 'deployed-server' as well.

Lastly, there is container work ongoing for the Overcloud. Again, I'd
like to see us adopt a format that would allow it to be used in the
Undercloud as well as opposed to having to re-implement features in the
Over and Under clouds all the time.

Undercloud installation is already sometimes fragile, but it's
probably 
the least fragile part right now (at least from my experience) And
at 
the very least it's pretty obviously debuggable in most cases. THT
is 
hard to understand and often impossible to debug. I'd prefer we move 
away from THT completely rather than trying to fix it in one more
place 
where heat does not fit..

What tool did you have in mind. FWIW I started with heat because by
using just Heat I was able to take the initial steps to prototype this.

In my mind Mistral might be next here and in fact it already supports
the single process launching idea thing. Keeping the undercloud
installer as light as possible would be ideal though.

Dan

I created a short video demonstration which goes over some of the
history behind the approach, and shows a live demo of all of this
working with the patches above:

https://www.youtube.com/watch?v=y1qMDLAf26Q

Thoughts? Would it be cool to have a session to discuss this more
in
Barcelona?

Dan Prince (dprince)



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubs
cribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dan_Prince (8,160 points)   1 5 8
0 votes

On Thu, Aug 4, 2016 at 11:48 PM, Dan Prince dprince@redhat.com wrote:
Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL.

I saw those patches coming, I'm interested in the all-in-one approach,
if only for testing purpose. I hope to be able to propose a solution
with broker-less RPC instead of fake RPC at some point, but it's a
good first step.

I'm a bit more intrigued by the no-auth patch. It seems that Heat
would rely heavily on Keystone interactions even after initial
authentication, so I wonder how that work. As it seems you would need
to push the same approach to Ironic, have you considered starting
Keystone instead? It's a simple WSGI service, and can work with SQLite
as well I believe.

--
Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by therve_at_redhat.com (2,620 points)   1 3
0 votes

On 08/05/2016 01:21 PM, Steven Hardy wrote:
On Fri, Aug 05, 2016 at 12:27:40PM +0200, Dmitry Tantsur wrote:

On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.

I don't want to sound rude, but please no. The fact that you have a hammer
does not mean everything around is nails :( What problem are you trying to
solve by doing it?

I think Dan explains it pretty well in his video, and your comment
indicates a fundamental misunderstanding around the entire TripleO vision,
which is about symmetry and reuse between deployment tooling and the
deployed cloud.

Well, except for you need some non-openstack starting point, because
unlike with e.g. ansible installing any openstack service(s) does not
end at "dnf install ".

The problems this would solve are several:

  1. Remove divergence between undercloud and overcloud puppet implementation
    (instead of having an undercloud specific manifest, we reuse the exact
    same stuff we use for overcloud deployments)

I'm not against reusing puppet bits, I'm against building the same heavy
abstraction layer with heat around it.

  1. Better modularity, far easier to enable/disable services

Why? Do you expect enabling/disabling Nova, for example? In this regard
undercloud is fundamentally different from overcloud: for the former we
have a list of required services and a pretty light list of optional
services.

  1. Get container integration "for free" when we land it in the overcloud

  2. Any introspection and debugging workflow becomes identical between the
    undercloud and overcloud

I would love a defined debugging workflow for the overcloud first..

  1. We remove dependencies on a bunch of legacy scripts which run outside of
    puppet

If you mean instack-undercloud element, we're getting rid of them
anyway, no?

  1. Whenever someone lands support for a new service in the overcloud, we
    automatically get undercloud support for it, completely for free.

Again, why? A service won't integrate itself into the deployment. And to
be honest, the amount of options TripleO has already cases real world
problems. I would rather see a well defined set of functionality for it..

  1. Potential for much easier implementation of a multi-node undercloud

Ideally, I would love to see:

for node in nodes:
ssh $node puppet apply blah-blah

Maybe we're not there, but it only means we have to improve our puppet
modules.

Undercloud installation is already sometimes fragile, but it's probably the
least fragile part right now (at least from my experience) And at the very
least it's pretty obviously debuggable in most cases. THT is hard to
understand and often impossible to debug. I'd prefer we move away from THT
completely rather than trying to fix it in one more place where heat does
not fit..

These are some strong but unqualified assertions, so it's hard to really
respond.

We'll talk about "unqualified" assertions the next time I'll try to get
answers on #tripleo after seeing error messages like "controller_step42
failed with code 1" ;)

Yes, there is complexity, but it's a set of abstractions which
actually work pretty well for us, so there is value in having just one set
of abstractions used everywhere vs special-casing the undercloud.

There should be a point where we stop. What entity is going to install
heat to install undercloud (did I just say "seed")? What will provide HA
for it? Authentication, templates storage and versioning? How do you
reuse the same abstractions (that's the whole point after all)?

Re moving away from THT completely, this is not a useful statement -
yes, there are alternative tools, but if you were to remove THT and just
use some other tool with Ironic, the result would simply not be TripleO.
There would be zero migration/upgrade path for existing users and all
third-party integrations (and our API/UI) would break.

I don't agree it would not be TripleO. OpenStack does not end on heat
templates, some deployments don't even use heat.

FWIW I think this prototyping work is very interesting, and I'm certainly
keen to get wider (more constructive) feedback and see where it leads.

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dmitry_Tantsur (18,080 points)   2 4 8
0 votes

On 08/05/2016 01:34 PM, Dan Prince wrote:
On Fri, 2016-08-05 at 12:27 +0200, Dmitry Tantsur wrote:

On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new way
to
install the Undercloud. The driving force behind this was some of
the
recent "composable services" work we've done in TripleO so
initially I
called in composable undercloud. There is an etherpad here with
links
to some of the patches already posted upstream (many of which stand
as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process
all-
in-one heat-all (engine and API) and thereby avoid things like
Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.
I don't want to sound rude, but please no. The fact that you have a
hammer does not mean everything around is nails :( What problem are
you
trying to solve by doing it?

Several problems I think.

One is TripleO has gradually moved away from elements. And while we
still use DIB elements for some things we no longer favor that tool and
instead rely on Heat and config management tooling to do our stepwise
deployment ordering. This leaves us using instack-undercloud a tool
built specifically to install elements locally as a means to create our
undercloud. It works... and I do think we've packaged it nicely but it
isn't the best architectural fit for where we are going I think. I
actually think that from an end/user contribution standpoint using t-h-
t could be quite nice for adding features to the Undercloud.

I don't quite get how it is better than finally moving to puppet only
and stop using elements.

Second would be re-use. We just spent a huge amount of time in Newton
(and some in Mitaka) refactoring t-h-t around composable services. So
say you add a new composable service for Barbican in the Overcloud...
wouldn't it be nice to be able to consume the same thing in your
Undercloud as well? Right now you can't, you have to do some of the
work twice and in quite different formats I think. Sure, there is some
amount of shared puppet work but that is only part of the picture I
think.

I've already responded to Steve's email, so a tl;dr here: I'm not sure
why you want to add random services to undercloud. Have you seen an
installer ever benefiting from e.g. adding a FileSystem-as-a-Service or
Database-as-a-Service solution?

There are new features to think about here too. Once upon a time
TripleO supported multi-node underclouds. When we switched to instack-
undercloud we moved away from that. By switching back to tripleo-heat-
templates we could structure our templates around abstractions like
resource groups and the new 'deployed-server' trick that allow you to
create machines either locally or perhaps via Ironic too. We could
avoid Ironic entirely and always install the Undercloud on existing
servers via 'deployed-server' as well.

A side note: if we do use Ironic for this purpose, I would expect some
help with pushing the Ironic composable service through. And the
ironic-inspector's one, which I haven't even started.

I'm still struggling to understand what entity is going to install this
bootstrapping Heat instance. Are we bringing back seed?

Lastly, there is container work ongoing for the Overcloud. Again, I'd
like to see us adopt a format that would allow it to be used in the
Undercloud as well as opposed to having to re-implement features in the
Over and Under clouds all the time.

Undercloud installation is already sometimes fragile, but it's
probably
the least fragile part right now (at least from my experience) And
at
the very least it's pretty obviously debuggable in most cases. THT
is
hard to understand and often impossible to debug. I'd prefer we move
away from THT completely rather than trying to fix it in one more
place
where heat does not fit..

What tool did you have in mind. FWIW I started with heat because by
using just Heat I was able to take the initial steps to prototype this.

In my mind Mistral might be next here and in fact it already supports
the single process launching idea thing. Keeping the undercloud
installer as light as possible would be ideal though.

I don't have a really huge experience with both, but for me Mistral
seems much cleaner and easier to understand. That, of course, won't
allow you to use reuse the existing heat templates (which may be good or
bad depending on your point of view).

Dan

I created a short video demonstration which goes over some of the
history behind the approach, and shows a live demo of all of this
working with the patches above:

https://www.youtube.com/watch?v=y1qMDLAf26Q

Thoughts? Would it be cool to have a session to discuss this more
in
Barcelona?

Dan Prince (dprince)



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubs
cribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dmitry_Tantsur (18,080 points)   2 4 8
0 votes

On Fri, 2016-08-05 at 13:39 +0200, Thomas Herve wrote:
On Thu, Aug 4, 2016 at 11:48 PM, Dan Prince dprince@redhat.com
wrote:

Last week I started some prototype work on what could be a new way
to
install the Undercloud. The driving force behind this was some of
the
recent "composable services" work we've done in TripleO so
initially I
called in composable undercloud. There is an etherpad here with
links
to some of the patches already posted upstream (many of which stand
as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process
all-
in-one heat-all (engine and API) and thereby avoid things like
Rabbit,
and MySQL.
I saw those patches coming, I'm interested in the all-in-one
approach,
if only for testing purpose. I hope to be able to propose a solution
with broker-less RPC instead of fake RPC at some point, but it's a
good first step.

I'm a bit more intrigued by the no-auth patch. It seems that Heat
would rely heavily on Keystone interactions even after initial
authentication, so I wonder how that work. As it seems you would need
to push the same approach to Ironic, have you considered starting
Keystone instead? It's a simple WSGI service, and can work with
SQLite
as well I believe.

You are correct. Noauth wasn't enough. I had to add a bit more to make
OS::Heat::SoftwareDeployments happy to get the templates I showed in
the demo working. Surprisingly though if I avoid Heat
OS::Heat::SoftwareDeployments and only used OS:Heat::SoftwareConfig's
in my templates no extra keystone auth was needed. This is because heat
only creates the extra Keystone user, trust, etc. when realizing the
software deployments I think.

I started with this which should work for multiple projects besides
just Heat: https://review.openstack.org/#/c/351351/2/tripleoclient/fake
_keystone.py

I'd be happy to swap in full Keystone if people prefer but that would
be more memory, and setup. Keystone dropped it's eventlet runner
recently so we'd have to fork another WSGI process to run it I think
somewhere in an out of the way (non-default ports, etc) fashion. I was
trying to keep the project list minimal so I went and stubbed in only
what was functionally needed for this here with an eye that we'd
actually (at some point) make heat support true noauth again.

Dan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dan_Prince (8,160 points)   1 5 8
0 votes

Hi,

On Fri, Aug 5, 2016 at 7:56 PM, Dmitry Tantsur dtantsur@redhat.com wrote:
Ideally, I would love to see:

for node in nodes:
ssh $node puppet apply blah-blah

Maybe we're not there, but it only means we have to improve our puppet
modules.

This is the same thought I had, Shouldn't the config be just a call to
a manifest?

The undercloud should be a simple install process with some config (or
image based deployment). Using Heat to deploy the undercloud means
involves bootstrapping a Heat environment. I believe Ansible feels
like a much better fit for this. What would the user/administrator
want? Is customization of the undercloud something realistically
happening?

regards,

Gerard

--

Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Gerard_Braad (1,400 points)   3
0 votes

On Fri, Aug 05, 2016 at 01:56:32PM +0200, Dmitry Tantsur wrote:
On 08/05/2016 01:21 PM, Steven Hardy wrote:

On Fri, Aug 05, 2016 at 12:27:40PM +0200, Dmitry Tantsur wrote:

On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new way to
install the Undercloud. The driving force behind this was some of the
recent "composable services" work we've done in TripleO so initially I
called in composable undercloud. There is an etherpad here with links
to some of the patches already posted upstream (many of which stand as
general imporovements on their own outside the scope of what I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single process all-
in-one heat-all (engine and API) and thereby avoid things like Rabbit,
and MySQL. Then we can use Heat templates to drive the Undercloud
deployment just like we do in the Overcloud.

I don't want to sound rude, but please no. The fact that you have a hammer
does not mean everything around is nails :( What problem are you trying to
solve by doing it?

I think Dan explains it pretty well in his video, and your comment
indicates a fundamental misunderstanding around the entire TripleO vision,
which is about symmetry and reuse between deployment tooling and the
deployed cloud.

Well, except for you need some non-openstack starting point, because unlike
with e.g. ansible installing any openstack service(s) does not end at "dnf
install ".

You might like to watch Dan's demo again.

It goes something like:

yum install python-tripleoclient
openstack undercloud deploy

Done!

The problems this would solve are several:

  1. Remove divergence between undercloud and overcloud puppet implementation
    (instead of having an undercloud specific manifest, we reuse the exact
    same stuff we use for overcloud deployments)

I'm not against reusing puppet bits, I'm against building the same heavy
abstraction layer with heat around it.

Sure, this is a valid concern to raise, and analternative to what Dan has
prototyped would be to refactor the undercloud puppet manifest to use the
puppet-tripleo profiles, somebody still has to do this work and it still
doesn't help at all with either container integration or multi-node
underclouds.

  1. Better modularity, far easier to enable/disable services

Why? Do you expect enabling/disabling Nova, for example? In this regard
undercloud is fundamentally different from overcloud: for the former we have
a list of required services and a pretty light list of optional services.

Actually, yes! I'd love to be able to disable Nova and instead deploy
nodes directly via a mistral workflow that drives Ironic. That's why I
started this:

https://review.openstack.org/#/c/313048/

There are reasons such as static IPs for everything where you might want to
be able to make Neutron optional, and there are already a bunch of optional
services (such as all the telemetry services).

Ok, every time I want to disable or add a new service I can hack on the
manifest, but it's just extra work compared to reusing the exact same
method we already support for overcloud deployments.

  1. Get container integration "for free" when we land it in the overcloud

  2. Any introspection and debugging workflow becomes identical between the
    undercloud and overcloud

I would love a defined debugging workflow for the overcloud first..

Sure, and it's something we have to improve regardless.

  1. We remove dependencies on a bunch of legacy scripts which run outside of
    puppet

If you mean instack-undercloud element, we're getting rid of them anyway,
no?

Quite a few still remain, but yeah there are less than there was, which is
good.

  1. Whenever someone lands support for a new service in the overcloud, we
    automatically get undercloud support for it, completely for free.

Again, why? A service won't integrate itself into the deployment. And to be
honest, the amount of options TripleO has already cases real world problems.
I would rather see a well defined set of functionality for it..

It means it's easy to enable any service which is one less barrier to
integration, I'm not really sure how that could be construed as a bad
thing.

  1. Potential for much easier implementation of a multi-node undercloud

Ideally, I would love to see:

for node in nodes:
ssh $node puppet apply blah-blah

Haha, this is a delightful over-simplification, but it completely ignores
all of the logic to create the per-node manifests and hieradata. This is
what the Heat templates already do for us, over multiple nodes by default.

Maybe we're not there, but it only means we have to improve our puppet
modules.

There is a layer of orchestration outside of the per-service modules which
is needed here. We do that simply in the current undercloud implementation
by having a hard-coded manifest, which works OK. We do that in the
overcloud by orchestrating puppet via Heat over multiple nodes, which also
works OK.

Undercloud installation is already sometimes fragile, but it's probably the
least fragile part right now (at least from my experience) And at the very
least it's pretty obviously debuggable in most cases. THT is hard to
understand and often impossible to debug. I'd prefer we move away from THT
completely rather than trying to fix it in one more place where heat does
not fit..

These are some strong but unqualified assertions, so it's hard to really
respond.

We'll talk about "unqualified" assertions the next time I'll try to get
answers on #tripleo after seeing error messages like "controller_step42
failed with code 1" ;)

Next time that happens, try typing "openstack stack failures list
overcloud" - it's a first step to improving the way we surface errors.

Yes, there is complexity, but it's a set of abstractions which
actually work pretty well for us, so there is value in having just one set
of abstractions used everywhere vs special-casing the undercloud.

There should be a point where we stop. What entity is going to install heat
to install undercloud (did I just say "seed")? What will provide HA for it?
Authentication, templates storage and versioning? How do you reuse the same
abstractions (that's the whole point after all)?

Watch the demo. Tripleoclient runs a heat process, which drives
applying puppet. There is no seed, but I guess it could be a disposable
container (as Dan said in his demo).

Thanks,

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Steven_Hardy (16,900 points)   2 7 12
0 votes

On Fri, 2016-08-05 at 13:56 +0200, Dmitry Tantsur wrote:
On 08/05/2016 01:21 PM, Steven Hardy wrote:

On Fri, Aug 05, 2016 at 12:27:40PM +0200, Dmitry Tantsur wrote:

On 08/04/2016 11:48 PM, Dan Prince wrote:

Last week I started some prototype work on what could be a new
way to
install the Undercloud. The driving force behind this was some
of the
recent "composable services" work we've done in TripleO so
initially I
called in composable undercloud. There is an etherpad here with
links
to some of the patches already posted upstream (many of which
stand as
general imporovements on their own outside the scope of what
I'm
talking about here).

https://etherpad.openstack.org/p/tripleo-composable-undercloud

The idea in short is that we could spin up a small single
process all-
in-one heat-all (engine and API) and thereby avoid things like
Rabbit,
and MySQL. Then we can use Heat templates to drive the
Undercloud
deployment just like we do in the Overcloud.
I don't want to sound rude, but please no. The fact that you have
a hammer
does not mean everything around is nails :( What problem are you
trying to
solve by doing it?
I think Dan explains it pretty well in his video, and your comment
indicates a fundamental misunderstanding around the entire TripleO
vision,
which is about symmetry and reuse between deployment tooling and
the
deployed cloud.
Well, except for you need some non-openstack starting point, because 
unlike with e.g. ansible installing any openstack service(s) does
not 
end at "dnf install ".

The problems this would solve are several:

  1. Remove divergence between undercloud and overcloud puppet
    implementation
    (instead of having an undercloud specific manifest, we reuse the
    exact
    same stuff we use for overcloud deployments)
    I'm not against reusing puppet bits, I'm against building the same
    heavy 
    abstraction layer with heat around it.

What do you mean by heavy exactly. The entire point here was to
demonstrate that this can work and is actually quite lightweight I
think.

We are already building an abstraction layer. So why not just use it in
2 places instead of one.

  1. Better modularity, far easier to enable/disable services
    Why? Do you expect enabling/disabling Nova, for example? In this
    regard 
    undercloud is fundamentally different from overcloud: for the former
    we 
    have a list of required services and a pretty light list of optional 
    services.

I think this is a very narrow view of the Undercloud and ignores the
fact that continually adding booleans to enable or disable features is
not scalable. Using the same composability and deployment framework we
have developed for the Overcloud might make better sense to me.

There is also real potential here to re-use this as a means to install
other package based types of setups. An "anything is an undercloud"
sort of approach could be the next logic step... all of this for free
because we are building abstractions to install these things in the
Overcloud as well.

  1. Get container integration "for free" when we land it in the
    overcloud

  2. Any introspection and debugging workflow becomes identical
    between the
    undercloud and overcloud
    I would love a defined debugging workflow for the overcloud first..

The nice thing about demo I showed for debugging is all the output
comes back to the console. Heat, os-collect-config, puppet, etc. all
there at your fingertips. Set 'debug=True' and you have everything you
need I think.

After building it I've quite enjoyed how fast it is to test and debug
creating a prototype undercloud.yaml.

  1. We remove dependencies on a bunch of legacy scripts which run
    outside of
    puppet
    If you mean instack-undercloud element, we're getting rid of them 
    anyway, no?

We mean all of the elements. Besides a few bootstrapping things we have
gradually moved towards using Heat hooks to run things as opposed to
the traditional os-apply-config/os-refresh-config hooks. This provides
better signalling back to heat and arguably makes debugging much easier
when something fails too.

  1. Whenever someone lands support for a new service in the
    overcloud, we
    automatically get undercloud support for it, completely for free.
    Again, why? A service won't integrate itself into the deployment. And
    to 
    be honest, the amount of options TripleO has already cases real
    world 
    problems. I would rather see a well defined set of functionality for
    it..

  2. Potential for much easier implementation of a multi-node
    undercloud
    Ideally, I would love to see:

  for node in nodes:
    ssh $node puppet apply blah-blah

Maybe we're not there, but it only means we have to improve our
puppet 
modules.

Undercloud installation is already sometimes fragile, but it's
probably the
least fragile part right now (at least from my experience) And at
the very
least it's pretty obviously debuggable in most cases. THT is hard
to
understand and often impossible to debug. I'd prefer we move away
from THT
completely rather than trying to fix it in one more place where
heat does
not fit..
These are some strong but unqualified assertions, so it's hard to
really
respond.
We'll talk about "unqualified" assertions the next time I'll try to
get 
answers on #tripleo after seeing error messages like
"controller_step42 
failed with code 1" ;)

Yes, there is complexity, but it's a set of abstractions which
actually work pretty well for us, so there is value in having just
one set
of abstractions used everywhere vs special-casing the undercloud.
There should be a point where we stop. What entity is going to
install 
heat to install undercloud (did I just say "seed")? What will provide
HA 
for it? Authentication, templates storage and versioning? How do you 
reuse the same abstractions (that's the whole point after all)?

Re moving away from THT completely, this is not a useful statement
-
yes, there are alternative tools, but if you were to remove THT and
just
use some other tool with Ironic, the result would simply not be
TripleO.
There would be zero migration/upgrade path for existing users and
all
third-party integrations (and our API/UI) would break.
I don't agree it would not be TripleO. OpenStack does not end on
heat 
templates, some deployments don't even use heat.

FWIW I think this prototyping work is very interesting, and I'm
certainly
keen to get wider (more constructive) feedback and see where it
leads.

Thanks,

Steve



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubs
cribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 5, 2016 by Dan_Prince (8,160 points)   1 5 8
...