settingsLogin | Registersettings

[Openstack-operators] Case studies on Openstack HA architecture

0 votes

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ cluster and 3 Controllers. I am wondering whether there are any studies done that show the pros and cons of co-locating database and messaging service with the Openstack control services. In other words, I am very interested in learning about advantages and disadvantages, in terms of ease of deployment, upgrade and overall API performance, of having 3 all-in-one Openstack controller over a more distributed deployment model.

References to any work done in this area will be highly appreciated.

Thanks,
Imtiaz


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Aug 30, 2017 in openstack-operators by Imtiaz_Chowdhury (180 points)  

8 Responses

0 votes

I'm not aware of any studies as per se, but we have long run rabbitmq,
MySQL, and all the API endpoints on the same three nodes.

On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury" imtiaz.chowdhury@workday.com
wrote:

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
cluster and 3 Controllers. I am wondering whether there are any studies
done that show the pros and cons of co-locating database and messaging
service with the Openstack control services. In other words, I am very
interested in learning about advantages and disadvantages, in terms of ease
of deployment, upgrade and overall API performance, of having 3 all-in-one
Openstack controller over a more distributed deployment model.

References to any work done in this area will be highly appreciated.

Thanks,
Imtiaz


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 26, 2017 by David_Medberry (8,000 points)   1 4 5
0 votes

We've also always done the same. We deploy any cloud with a minimum of 3 (quorum, etc) but when the database or message queue needs to scale up, the other components probably need to as well so making an equal replica of everything across the environment.

Sent from my iPhone

On Aug 26, 2017, at 9:13 AM, David Medberry openstack@medberry.net wrote:

I'm not aware of any studies as per se, but we have long run rabbitmq, MySQL, and all the API endpoints on the same three nodes.

On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury" imtiaz.chowdhury@workday.com wrote:
Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ cluster and 3 Controllers. I am wondering whether there are any studies done that show the pros and cons of co-locating database and messaging service with the Openstack control services. In other words, I am very interested in learning about advantages and disadvantages, in terms of ease of deployment, upgrade and overall API performance, of having 3 all-in-one Openstack controller over a more distributed deployment model.

References to any work done in this area will be highly appreciated.

Thanks,
Imtiaz


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 26, 2017 by Mohammed_Naser (3,860 points)   1 3
0 votes

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ cluster and 3 Controllers.
I am wondering whether there are any studies done that show the pros and cons of co-locating database and messaging service with the Openstack control services.
In other words, I am very interested in learning about advantages and disadvantages, in terms of ease of deployment,
upgrade and overall API performance, of having 3 all-in-one Openstack controller over a more distributed deployment model.

In general, a host with less running services is easier to troubleshoot.
However, there is no fundamental issue with sharing mysql/rabbit on a single controller assuming the host is quick enough.

For the OpenStack API services I would highly recommend “splitting” things up.
It does not necessarily need to be on different physical hosts but at least use something to prevent package dependencies between the different OpenStack components (virtualenv/containers/vms).
If you do not do this it makes upgrades of an individual component on a node near impossible.

Cheers,
Robert van Leeuwen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 28, 2017 by Van_Leeuwen,_Robert (1,740 points)   3
0 votes

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
imtiaz.chowdhury@workday.com wrote:
Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
cluster and 3 Controllers. I am wondering whether there are any studies done
that show the pros and cons of co-locating database and messaging service
with the Openstack control services. In other words, I am very interested
in learning about advantages and disadvantages, in terms of ease of
deployment, upgrade and overall API performance, of having 3 all-in-one
Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.

References to any work done in this area will be highly appreciated.

Thanks,
Imtiaz


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Blog: serverascode.com


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 28, 2017 by Curtis (4,180 points)   2 3
0 votes

Thanks Curtis, Robert, David and Mohammed for your responses.

As a follow up question, do you use any deployment automation tools for setting up the HA control plane? I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS.

Imtiaz

On 8/28/17, 2:23 PM, "Curtis" serverascode@gmail.com wrote:

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
<imtiaz.chowdhury@workday.com> wrote:

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
cluster and 3 Controllers. I am wondering whether there are any studies done
that show the pros and cons of co-locating database and messaging service
with the Openstack control services. In other words, I am very interested
in learning about advantages and disadvantages, in terms of ease of
deployment, upgrade and overall API performance, of having 3 all-in-one
Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.
>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA&m=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk&s=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc&e= 
>
-- 
Blog: serverascode.com


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 29, 2017 by Imtiaz_Chowdhury (180 points)  
0 votes

kolla has various containerization tools. one based on ansible, another based on kubernetes.


From: Imtiaz Chowdhury [imtiaz.chowdhury@workday.com]
Sent: Monday, August 28, 2017 5:24 PM
To: Curtis
Cc: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] Case studies on Openstack HA architecture

Thanks Curtis, Robert, David and Mohammed for your responses.

As a follow up question, do you use any deployment automation tools for setting up the HA control plane? I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS.

Imtiaz

On 8/28/17, 2:23 PM, "Curtis" serverascode@gmail.com wrote:

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
<imtiaz.chowdhury@workday.com> wrote:

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
cluster and 3 Controllers. I am wondering whether there are any studies done
that show the pros and cons of co-locating database and messaging service
with the Openstack control services. In other words, I am very interested
in learning about advantages and disadvantages, in terms of ease of
deployment, upgrade and overall API performance, of having 3 all-in-one
Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.
>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA&m=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk&s=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc&e=
>
--
Blog: serverascode.com


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 29, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Thanks Curtis, Robert, David and Mohammed for your responses.
As a follow up question, do you use any deployment automation tools for setting up the HA control plane?
I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS.

Here we currently have a physical loadbalancer which provides the ALL the HA logic.
The services are installed on vms managed by Puppet.
IMHO a loadbalancer is the right place to solve HA since you only have to do it once.
(Depending on your Neutron implementation you might also need something for Neutron)

To rant a bit about deployment/automation:
I am not necessarily a fan of management with puppet since module dependencies can become a nightmare even with things like R10K.
e.g. you want to deploy a newer version of keystone, this requires a newer openstack-common puppet module.
Now you have a change that affects everything (other OpenStack puppet modules also use openstack-common) and might now need upgrades as well.

To solve these kind of things we are looking at containers and investigated two possible deployment scenario’s:
OpenStack helm (+containers build by kolla) which is pretty nice and uses k8s.
The problem is that it is still early days for both k8s and helm.
Things that stuck out most:
* Helm: Nice for a deployment from scratch.
Integration with our environment is a bit of a pain (e.g. if you want to start with just one service)
It would need a lot of work to above into the product and not everything would be easy to get into upstream.
Still a very early implementation needs quite a bit of TLC.
If you can live with what comes out of the box it might be a nice solution.
* K8S: It is a relatively complex product and it is still missing some features especially for self-hosted installations.

After some deliberation, we decided to go with the “hashi” stack (with kola build containers).
This stack has more of a unix philosophy, simple processes that do 1 thing well:
* Nomad –scheduling
* Consul - service discovery and KV store
* Vault – secret management
* Fabio zeroconf loadbalancer which integrates with Consul
In general this stack is really easy to understand for everyone. (just work with it half a day and you really understand what is going on under the hood)
There are no overlay networks :)
Lots of the stuff can break without much impact. E.g. Nomad is only relevant when you want to start/stop containers it can crash or turned off the rest of the time.
Another pro for this is that there is a significant amount of knowledge around these products in house.

To give an example on complexity: if you look at deployment of k8s itself and the hashi stack:
* deployment of k8s with kargo: you have a very large playblook which takes 30 minutes to run to setup a cluster.
* deployment of the hashi stuff: is just one binary for each component with 1 config file basically done in a few minutes if even that.

Cheers,
Robert van Leeuwen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 29, 2017 by Van_Leeuwen,_Robert (1,740 points)   3
0 votes

On Mon, Aug 28, 2017 at 6:24 PM, Imtiaz Chowdhury
imtiaz.chowdhury@workday.com wrote:
Thanks Curtis, Robert, David and Mohammed for your responses.

As a follow up question, do you use any deployment automation tools for setting up the HA control plane? I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS.

I think it's safe to say that all "distros" and "installers" for
openstack will converge on using containers of some kind. As to what
and how that is actually done, there are a lot of opinions, and I
think you would have to perform your own due diligence around what
makes sense to you in terms of technologies and implementations. I
don't think there is a right answer. :)

To me it seems to be boiling down to: 1) Kubernetes is overkill (and
use something simpler) or 2) If you don't use Kubernetes then you will
just end up building it yourself 1. I don't think it's actually that
simple, but those are the talking points.

I think, as has been mentioned in this thread so far, the major
opensource openstack "installers" or "managers" (?) are
openstack-ansible, kolla, tripleo, and now openstack-helm. Not sure if
I'm missing any, sorry if I did.

Frankly I find this all pretty interesting. :)

Thanks,
Curtis.

Imtiaz

On 8/28/17, 2:23 PM, "Curtis" serverascode@gmail.com wrote:

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
<imtiaz.chowdhury@workday.com> wrote:
> Hi Openstack operators,
>
>
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
> cluster and 3 Controllers. I am wondering whether there are any studies done
> that show the pros and cons of co-locating database and messaging service
> with the Openstack control services.  In other words, I am very interested
> in learning about advantages and disadvantages, in terms of ease of
> deployment, upgrade and overall API performance, of having 3 all-in-one
> Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.

>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA&m=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk&s=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc&e=
>



--
Blog: serverascode.com

--
Blog: serverascode.com


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Aug 30, 2017 by Curtis (4,180 points)   2 3
...