settingsLogin | Registersettings

[openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

0 votes

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My hopes might be a
bit naive shrugs

Flavio

[0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Jul 18, 2017 in openstack-dev by Flavio_Percoco (36,960 points)   3 8 11

35 Responses

0 votes

On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco flavio@redhat.com wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools
and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While
I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects,
I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially
now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example, in
the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts
(I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally
these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be isolated
and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these
roles
and run them using the existing docker-cmd implementation that is coming out
in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion
and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible
is
a known, powerfull, tool that has been adopted by many operators already.
It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate
concerns
in the deployment workflow and the idea of making it simple for users of
TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would
also
be useful for others and that we all will benefit from it... My hopes might
be a
bit naive shrugs

Of course I'm biased since I've been (a little) involved in that work
but I like the idea of :

  • Moving forward with our containerization. docker-cmd will help us
    for sure for this transition (I insist on the fact TripleO is a
    product that you can upgrade and we try to make it smooth for our
    operators), so we can't just trash everything and switch to a new
    tool. I think the approach that we're taking is great and made of baby
    steps where we try to solve different problems.
  • Using more Ansible - the right way - when it makes sense : with the
    TripleO containerization, we only use Puppet for Configuration
    Management, managing a few resources but not for orchestration (or not
    all the features that Puppet provide) and for Data Binding (Hiera). To
    me, it doesn't make sense for us to keep investing much in Puppet
    modules if we go k8s & Ansible. That said, see the next point.
  • Having a transition path between TripleO with Puppet and TripleO
    with apbs and have some sort of binding between previous hieradata
    generated by TripleO & a similar data binding within Ansible playbooks
    would help. I saw your PoC Flavio, I found it great and I think we
    should make https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/hiera.yaml
    optional when running apbs, and allow to provide another format (more
    Ansiblish) to let folks not using TripleO to use it. We also should
    target this new format and switch service by service in TripleO to use
    this new format, as long as apbs support both. I think that way we can
    step by step migrate to use Ansible for configuration management.

There are some things to figure out:
- We kind of found out solutions for OpenStack services - great - now
what do we do for services like MySQL, Apache, etc. Do we have
"standard" and "community-supported" apbs? Do we need to create some?
- Where the apbs should live? IMO in OpenStack and IMO not in big tent
for now, under no umbrella.
- Since we use Puppet modules which don't only make configuration
management but also some orchestration (like creating keystone
endpoints, etc) - where should we put this logic? +1 for apbs using
clean Ansible code (and not bash templating).
- How we can help our vendors to whom we asked them to write Puppet
modules to deploy their software (Contrail, Nuage, etc). We still
might need some sort or "running puppet from ansible" for some
software we wouldn't have apbs as quickly as we would need.

I hope I didn't divert the discussion but here's my feedback and food
for thoughts.
Thanks for your work,
--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by emilien_at_redhat.co (36,940 points)   2 6 10
0 votes

On 14.07.2017 11:17, Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,

It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])

which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if structured correctly, it'll allow
for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate
concerns
in the deployment workflow and the idea of making it simple for users of
TripleO
to do the same at runtime. Unfortunately, going down this road means
that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result
would also
be useful for others and that we all will benefit from it... My hopes
might be a
bit naive shrugs

Flavio

[0]
http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
On 14.07.2017 11:17, Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,

It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])

I agree it's hard to estimate but you gotta draw the line somewhere. I actually
spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote the
pyhelm lib (took some code from the openstack-helm folks) and I wrote the
ansible helm module myself. I'd say I've spent enough time on this research.

I don't think getting a full PoC working is worth it as that will require way
more work for not much value since we can anticipate some of the complexities
already.

As far as the complexity comment goes, I disagree with you. I don't think you're
evaluating the amount of complexity that there IS already in TripleO and how
adding more complexity (layers, states, services) would make things worse for
not much extra value.

By all means, I might be wrong here so, do let me know if you're seeing
something I'm not.
Flavio
--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 14, 2017 by Flavio_Percoco (36,960 points)   3 8 11
0 votes

On 14.7.2017 11:17, Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

I agree this is a direction we should explore further. This would give
us the option to tailor things exactly as we need -- good for keeping
our balance in having interfaces as stable as possible, while still
making enough development progress. And we'd keep our ability to make
important changes (e.g. bugfixes) without delays.

We'll have to write more code ourselves, but it's possible that if we
picked up an existing tool, we'd have to spend that time (if not more)
elsewhere. Migrating existing non-kubernetized TripleO deployments to
kubernetized is going to be pretty difficult even if we do what you
suggested. I imagine that if we also had to fit into some pre-existing
external deployment/management interfaces, while trying to keep ours
stable or make just iterative changes, it might turn out to be a surreal
effort. We will have to design things with migration from "legacy
TripleO" in mind, or make later amendments here and there solely for
this purpose. Such design and patches would probably not be a good fit
for non-tripleo projects.

What i recall from our old PoC [2], defining the resources and init
containers etc. will probably not be the most difficult task, and
furthermore we can largely draw inspiration from our current
containerized solution too. I think the more challenging things might be
e.g. config generation with Ansible, and how major upgrades and rolling
updates will be done (how all this ties into the APB way of
provisioning/deprovisioning). And of course how to fulfill the
expectations that TripleO has set around network isolation and HA :)

I'm eager to give the latest code a try myself :) Thanks for working on
this, it looks like there's been great progress lately!

Jirka

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My hopes might be a
bit naive shrugs

Flavio

[0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

[2] https://github.com/jistr/tripleo-kube-test

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by =?UTF-8?B?SmnFmcOtIF (3,860 points)   2 3
0 votes

Guys.... you just described Kolla-Kubernetes pretty much... how about
we join effort and work towards this goal together?

On 14 July 2017 at 08:43, Flavio Percoco flavio@redhat.com wrote:
On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:

On 14.07.2017 11:17, Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,

It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])

I agree it's hard to estimate but you gotta draw the line somewhere. I
actually
spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
the
pyhelm lib (took some code from the openstack-helm folks) and I wrote the
ansible helm module myself. I'd say I've spent enough time on this research.

I don't think getting a full PoC working is worth it as that will require
way
more work for not much value since we can anticipate some of the
complexities
already.

As far as the complexity comment goes, I disagree with you. I don't think
you're
evaluating the amount of complexity that there IS already in TripleO and
how
adding more complexity (layers, states, services) would make things worse
for
not much extra value.

By all means, I might be wrong here so, do let me know if you're seeing
something I'm not.
Flavio
--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Michał_Jastrzębski (9,220 points)   1 5 5
0 votes

On 14.07.2017 17:55, Michał Jastrzębski wrote:
Guys.... you just described Kolla-Kubernetes pretty much... how about
we join effort and work towards this goal together?

That's exactly that I'd like we all to do.

On 14 July 2017 at 08:43, Flavio Percoco flavio@redhat.com wrote:

On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:

On 14.07.2017 11:17, Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,

It's hard to estimate that complexity w/o having a PoC of such an
integration. We should come up with a final choice once we have it done.

My vote would go for investing engineering resources into solutions that
have problems already solved, even by the price of added complexity (but
that sort of depends...). Added complexity may be compensated with
removed complexity (like those client -> Mistral -> Heat -> Mistral ->
Ansible manipulations discussed in the mail thread mentioned below [0])

I agree it's hard to estimate but you gotta draw the line somewhere. I
actually
spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
the
pyhelm lib (took some code from the openstack-helm folks) and I wrote the
ansible helm module myself. I'd say I've spent enough time on this research.

I don't think getting a full PoC working is worth it as that will require
way
more work for not much value since we can anticipate some of the
complexities
already.

As far as the complexity comment goes, I disagree with you. I don't think
you're
evaluating the amount of complexity that there IS already in TripleO and
how
adding more complexity (layers, states, services) would make things worse
for
not much extra value.

By all means, I might be wrong here so, do let me know if you're seeing
something I'm not.

My point was to "trade" complexity described in the "Forming our plans
around Ansible​" ML thread:

(3) Mistral calling Heat calling Mistral calling Ansible

to just

(3') something calls kolla-kubernetes/openstack-helm, via some wrapper
composition overlay (which creates complexity), or the like

While the latter might add complexity like the way you (Flavio) have
described, the former would remove another type of complexity, and the
result might worth the efforts.

Flavio
--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing more new k8s object deployment tooling?

There are efforts to make it easy to deploy kolla-kubernetes microservice charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object layer. Orchestration is done by ansible via TripleO, and the solutions already found/debugged to how to deploy OpenStack in containers on Kubernetes can be reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a tricky bit of code to get to work, but kolla-kubernetes has a solution to it. You can get it by: helm install kolla/keystone-fernet-rotate-job.

We designed this layer to be shareable so we all can contribute to the commons rather then having every project reimplement their own and have to chase bugs across all the implementations. The deployment projects will be stronger together if we can share as much as possible.

Please reconsider. I'd be happy to talk with you more if you want.

Thanks,
Kevin


From: Flavio Percoco [flavio@redhat.com]
Sent: Friday, July 14, 2017 2:17 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result would also
be useful for others and that we all will benefit from it... My hopes might be a
bit naive shrugs

Flavio

[0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

On 14.07.2017 18:16, Fox, Kevin M wrote:
https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing more new k8s object deployment tooling?

There are efforts to make it easy to deploy kolla-kubernetes microservice charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object layer. Orchestration is done by ansible via TripleO, and the solutions already found/debugged to how to deploy OpenStack in containers on Kubernetes can be reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a tricky bit of code to get to work, but kolla-kubernetes has a solution to it. You can get it by: helm install kolla/keystone-fernet-rotate-job.

We designed this layer to be shareable so we all can contribute to the commons rather then having every project reimplement their own and have to chase bugs across all the implementations. The deployment projects will be stronger together if we can share as much as possible.

Thank you Kevin, this ^^ expresses my thoughts better than I could ever say.

Please reconsider. I'd be happy to talk with you more if you want.

Thanks,
Kevin

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

First and foremost I just realized that I forgot to tag kolla and openstack-helm
in the subject so, I apologize. I'm glad the subject was catchy enough to get
your attention.

Just want to raise here what I just mentioned on IRC:

It's late in EU so I shouldn't be here right now but, I do want to point out
that, as usual, I asked for feedback and clarifications from everyone in this
thread.

I'm not trying to re-invent the wheel. What's in my original email is my
conclusion based on a research I did across the different tools there are. I
can, of course, be wrong and I'd like you all to help us by providing feedback.

I'm not expecting sales pitches but I'd love to have a more technical discussion
on how we can, hopefully, make this work.

On 14/07/17 16:16 +0000, Fox, Kevin M wrote:
https://xkcd.com/927/

I don't think adopting helm as a dependency adds more complexity then writing more new k8s object deployment tooling?

There are efforts to make it easy to deploy kolla-kubernetes microservice charts using ansible for orchestration in kolla-kubernetes. See:
https://review.openstack.org/#/c/473588/
What kolla-kubernetes brings to the table is a tested/shared base k8s object layer. Orchestration is done by ansible via TripleO, and the solutions already found/debugged to how to deploy OpenStack in containers on Kubernetes can be reused/shared.

See for example:
https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/main.yaml

I don't see much by way of dealing with fernet token rotation. That was a tricky bit of code to get to work, but kolla-kubernetes has a solution to it. You can get it by: helm install kolla/keystone-fernet-rotate-job.

It's just a PoC, don't take the implementation as definitive.

We designed this layer to be shareable so we all can contribute to the commons rather then having every project reimplement their own and have to chase bugs across all the implementations. The deployment projects will be stronger together if we can share as much as possible.

Please reconsider. I'd be happy to talk with you more if you want.

Let's talk, that's the whole point of this thread.
Flavio

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 14, 2017 by Flavio_Percoco (36,960 points)   3 8 11
0 votes

Out of curiosity, since I keep on hearing/reading all the tripleo
discussions on how tripleo folks are apparently thinking/doing?
redesigning the whole thing to use ansible + mistral + heat, or ansible
+ kubernetes or ansible + mistral + heat + ansible (a second time!) or ...

Seeing all those kinds of questions and suggestions around what should
be used and why and how (and even this thread) makes me really wonder
who actually uses tripleo and can afford/understand such kinds of changes?

Does anyone?

If there are is there going to be an upgrade
path for there existing 'cloud/s' to whatever this solution is?

What operator(s) has the ability to do such a massive shift at this
point in time? Who are these 'mystical' operators?

All this has really peaked my curiosity because I am personally trying
to do that shift (not exactly the same solution...) and I know it is a
massive undertaking (that will take quite a while to get right) even for
a simple operator with limited needs out of openstack (ie godaddy); so I
don't really understand how the generic solution for all existing
tripleo operators can even work...

Flavio Percoco wrote:

Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,
which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if structured correctly, it'll allow
for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate
concerns
in the deployment workflow and the idea of making it simple for users of
TripleO
to do the same at runtime. Unfortunately, going down this road means
that my
hope of creating a field where we could collaborate even more with other
deployment tools will be a bit limited but I'm confident the result
would also
be useful for others and that we all will benefit from it... My hopes
might be a
bit naive shrugs

Flavio

[0]
http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html
[1] https://github.com/tripleo-apb/tripleo-apbs

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by harlowja_at_fastmail (16,200 points)   2 5 8
...