settingsLogin | Registersettings

[Openstack] [OpenStack] [keystone] How to make keystone highly available?

0 votes

Hello,

I am thinking about using the keystone as an authentication system but I am
afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if our primary keystone server dies
we have to elect a new primary keystoe server. Are there some tools which
can be used in HA deployment?

Any piece of advice will be valuable

Sincerely,
Alexandr


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Sep 19, 2016 in openstack by Alexandr_Porunov (2,480 points)   2 14 18

16 Responses

0 votes

There is already ha options with pacemaker and corosync. Have you looked at that?

Inviato da iPhone

Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov alexandr.porunov@gmail.com ha scritto:

Hello,

I am thinking about using the keystone as an authentication system but I am afraid about failures which can affect all the cluster. In fact if the keystone server dies then our full cluster will stop. It would be better if we could use HA with the keystone. Then if our primary keystone server dies we have to elect a new primary keystoe server. Are there some tools which can be used in HA deployment?

Any piece of advice will be valuable

Sincerely,
Alexandr
!DSPAM:1,57e036cd282281177211101!


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

!DSPAM:1,57e036cd282281177211101!


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2016 by Remo_Mattei (10,500 points)   1 2 4
0 votes

Thank you for pointing on it. I haven't yet use pacemaker but I will try to
configure it.

Sincerely,
Alexandr

On Mon, Sep 19, 2016 at 10:23 PM, Remo Mattei Remo@italy1.com wrote:

There is already ha options with pacemaker and corosync. Have you looked
at that?

Inviato da iPhone

Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov <
alexandr.porunov@gmail.com> ha scritto:

Hello,

I am thinking about using the keystone as an authentication system but I
am afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if our primary keystone server dies
we have to elect a new primary keystoe server. Are there some tools which
can be used in HA deployment?

Any piece of advice will be valuable

Sincerely,
Alexandr
!DSPAM:1,57e036cd282281177211101!


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack

!DSPAM:1,57e036cd282281177211101!


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2016 by Alexandr_Porunov (2,480 points)   2 14 18
0 votes

If you'd like to avoid the pain of configuring this manually, Helion
OpenStack has many HA features enabled by default:

http://docs.hpcloud.com/#3.x/helion/planning/high_availability.html

On Mon, Sep 19, 2016 at 2:10 PM, Alexandr Porunov
alexandr.porunov@gmail.com wrote:
Thank you for pointing on it. I haven't yet use pacemaker but I will try to
configure it.

Sincerely,
Alexandr

On Mon, Sep 19, 2016 at 10:23 PM, Remo Mattei Remo@italy1.com wrote:

There is already ha options with pacemaker and corosync. Have you looked
at that?

Inviato da iPhone

Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov
alexandr.porunov@gmail.com ha scritto:

Hello,

I am thinking about using the keystone as an authentication system but I
am afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if our primary keystone server dies
we have to elect a new primary keystoe server. Are there some tools which
can be used in HA deployment?

Any piece of advice will be valuable

Sincerely,
Alexandr
!DSPAM:1,57e036cd282281177211101!


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

!DSPAM:1,57e036cd282281177211101!


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 19, 2016 by John_van_Ommen (1,100 points)   1 1
0 votes

Excerpts from Alexandr Porunov's message of 2016-09-19 21:46:54 +0300:

Hello,

I am thinking about using the keystone as an authentication system but I am
afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if our primary keystone server dies
we have to elect a new primary keystoe server. Are there some tools which
can be used in HA deployment?

Any piece of advice will be valuable

Hi Alexandr.

Keystone is a stateless application. It stores all of the state in
a SQL database, or in files that are immutable (for Fernet token key
files). So, what you really want is an HA SQL solution, and (if you're
using Fernet tokens) a key sync mechanism.

There are many such standard solutions. If you're fine with one server
worth of capacity, then DRBD+Pacemaker+Corosync are a pretty simple
option to keep MySQL HA. Once you do that, you can just use a load
balancer or something like UCARP/VRRP to make sure HTTP requests arrive
at a working keystone node.

Good luck.

responded Sep 19, 2016 by Clint_Byrum (40,940 points)   4 5 9
0 votes

Hello,

I am thinking about using the keystone as an authentication system but I am afraid about failures which can affect all the cluster. In fact if the keystone server dies then our full cluster will stop. It would be better if we could use HA with the keystone. Then if > our primary keystone server dies we have to elect a new primary keystoe server. Are there some tools which can be used in HA deployment?

Any piece of advice will be valuable

IMHO: get a proper ha load-balancer solution, make sure you install at least 2 of all APIS and load-balance in active-active mode
The OpenStack APIs are stateless so you can run multiple servers at the same time as long as they can connect to the same database backend.

The load-balancer is usable for all APIs, you can use it as the HA-IP for mysql and can also be used to offload SSL so you just have one place to configure your certificates.
(for galera, make use of a “sorry-server” and not active-active load-balancing for writes)

The great thing about a load-balancer is that you remove all HA complexity out of the OpenStack setup.
In my personal opinion the pacemaker setups can function OK but pacemaker is a complex piece of software and it is not unlikely to cause downtime either to misconfiguration or inexperienced people operating it.
Especially if pacemaker is also starting/stopping mysql/rabbit/openstack services and not only moving around a few IPs.
If you are going the pacemaker way make sure you play around with it quite a bit and do failure tests so you are comfortable with the commands and know what to look for when things go wrong.
Do not forget: any other people operating the pacemaker cluster will need that knowledge and a 2 node cluster is not a real cluster (split-brains).

Cheers,
Robert van Leeuwen


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 20, 2016 by Van_Leeuwen,_Robert (1,740 points)   1 3
0 votes

Hello everyone,

Thank you all for your advice!
In my case one keystone server can easily hold a load. I don't need to
balance a load between two or more keystone servers. However I need two
keystone servers for high availability. So, I decided just to use two
keystone servers with the same virtual IP address.
Pros:
- I can easily configure HA
- I don't need to use a master/master replication for a database.
- I don't need to use load balancers.
- I don't need to learn peacemaker
Cons:
- It will not increase the overall throughput
- One of the keystone servers will always be at a cold state

However. We can rid of cons by using two virtual IP adresses. So, we can
use a half proxy servers with keystone1 and another half of proxy servers
with keystone2. If one of the keystone servers will be dead then keepalived
will assign both virtual ip addresses to the alive node. After node will be
brought back to live keepalived will reassign IP addresses back (It isn't
the load balancing but our throughput will increase). If our clients will
use proxy servers with both keystone1 and keystone 2 then we have to use
master/master replication for the database.

Best regards,
Alexandr

On Tue, Sep 20, 2016 at 8:49 AM, Van Leeuwen, Robert rovanleeuwen@ebay.com
wrote:

>

Hello,

I am thinking about using the keystone as an authentication system but I
am afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if > our primary keystone server
dies we have to elect a new primary keystoe server. Are there some tools
which can be used in HA deployment?

Any piece of advice will be valuable

IMHO: get a proper ha load-balancer solution, make sure you install at
least 2 of all APIS and load-balance in active-active mode

The OpenStack APIs are stateless so you can run multiple servers at the
same time as long as they can connect to the same database backend.

The load-balancer is usable for all APIs, you can use it as the HA-IP for
mysql and can also be used to offload SSL so you just have one place to
configure your certificates.

(for galera, make use of a “sorry-server” and not active-active
load-balancing for writes)

The great thing about a load-balancer is that you remove all HA complexity
out of the OpenStack setup.

In my personal opinion the pacemaker setups can function OK but pacemaker
is a complex piece of software and it is not unlikely to cause downtime
either to misconfiguration or inexperienced people operating it.

Especially if pacemaker is also starting/stopping mysql/rabbit/openstack
services and not only moving around a few IPs.

If you are going the pacemaker way make sure you play around with it quite
a bit and do failure tests so you are comfortable with the commands and
know what to look for when things go wrong.

Do not forget: any other people operating the pacemaker cluster will need
that knowledge and a 2 node cluster is not a real cluster (split-brains).

Cheers,

Robert van Leeuwen


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 20, 2016 by Alexandr_Porunov (2,480 points)   2 14 18
0 votes

On Sep 20, 2016, at 3:09 PM, Alexandr Porunov wrote:

So, I decided just to use two keystone servers with the same virtual IP address.

Now that you've made your decision, I'd like to ask some
followup questions for my own decision if I may.

Most everyone have talked about some kind of load balancer or
HA solution.

My setup is such that that wouldn't be .. "workable".

My whole Openstack setup is installed on a HP Bladecenter
with 16 half hight nodes so all communication etc is pretty
much internal in the blade center. I.e, the only external
link is the one to the rest of the flat and then to the
gatway/firewall/NATbox.

Also, having a LB in front of MySQL (and possibly one more
in front of Keystone), that will introduce a "single point
of failure" that I'd prefer not to have (even if it was
practically possible - I don't want to dedicate a whole
BC node just for that, they're to precious).

What would be the impact and/or problem by using DNS round
robin for MySQL and Keystone for example? And then have
MySQL in a master-master setup. I've never done that either,
so I'm not sure how good idea that would be.. Any pointers?

The only DB I've ever had the .. "displeasure" to try to get
to work in a master-master setup is OpenLDAP and I can remember
(vividly!) the discussions that's been on the OpenLDAP lists
about this over the years! In essence, "don't, for the love of
whoever, do it!!".
--
Choose a job you love, and you will never have
to work a day in your life.

responded Sep 20, 2016 by Turbo_Fredriksson (8,980 points)   7 12 15
0 votes

I think that I haven't understood your situation correctly but I will try
to suggest something:

If you have a big load on your keystone server you can spread the load
between two or more servers by using load balancers like HAProxy. But it
wouldn't work if keystone instances not shared tokens with each other. For
it you have to share databse to store all tokens which are generated by
your keystone servers. Personally I prefer MariaDB Galera Cluster because
it isn't hard to install and works very nice. To reach high availability
for load balancing you need to have one more load balancer managed by
keepalived.
(I am assuming that you don't have such a load)

If you care about high availability (as I do) then you need to have
additional keystone instance which will prevent your cluster from SPOF. For
it I use the same virtual IP address for both keystone instances managed by
keepalived.

Also you can use peacemaker and other stuff to reach high availability but
I can't give you advice for those tools because I haven't used them.

In your situation I would installed additional keystone instance in one of
your node and keepalived on both nodes to prevent SPOF.

Sincerely,
Alexandr

On Tue, Sep 20, 2016 at 10:56 PM, Turbo Fredriksson turbo@bayour.com
wrote:

On Sep 20, 2016, at 3:09 PM, Alexandr Porunov wrote:

So, I decided just to use two keystone servers with the same virtual IP
address.

Now that you've made your decision, I'd like to ask some
followup questions for my own decision if I may.

Most everyone have talked about some kind of load balancer or
HA solution.

My setup is such that that wouldn't be .. "workable".

My whole Openstack setup is installed on a HP Bladecenter
with 16 half hight nodes so all communication etc is pretty
much internal in the blade center. I.e, the only external
link is the one to the rest of the flat and then to the
gatway/firewall/NATbox.

Also, having a LB in front of MySQL (and possibly one more
in front of Keystone), that will introduce a "single point
of failure" that I'd prefer not to have (even if it was
practically possible - I don't want to dedicate a whole
BC node just for that, they're to precious).

What would be the impact and/or problem by using DNS round
robin for MySQL and Keystone for example? And then have
MySQL in a master-master setup. I've never done that either,
so I'm not sure how good idea that would be.. Any pointers?

The only DB I've ever had the .. "displeasure" to try to get
to work in a master-master setup is OpenLDAP and I can remember
(vividly!) the discussions that's been on the OpenLDAP lists
about this over the years! In essence, "don't, for the love of
whoever, do it!!".
--
Choose a job you love, and you will never have
to work a day in your life.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 20, 2016 by Alexandr_Porunov (2,480 points)   2 14 18
0 votes

On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote:

If you care about high availability (as I do) then you need to have
additional keystone instance which will prevent your cluster from SPOF.

That was the idea. One node is already dedicated for that, but I haven't
installed it yet, because I'm not sure what the right way to do this is.

I've been reading the high availability docs for the last couple of weeks
now, but they don't talk about the most basic things :(. For once, they
get right down to it (which I've complained about other docs don't in the
past :D).

IF (!) I understand things correctly, services report themselves into
the catalog (which is basically the *SQL server). This catalog is Keystone.

So my first Keystone server have registered itself as:

bladeA01:~# openstack endpoint list | grep keystone
| 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone | identity | True | internal | http://10.0.4.1:5000/v2.0 |
| 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone | identity | True | public | http://10.0.4.1:5000/v2.0 |
| c95ce66a4efa46b4855185d088279824 | europe-london | keystone | identity | True | admin | http://10.0.4.1:35357/v2.0 |

Now, I'm assuming that the second one will do the same (on ITS IP of
course). So "anyone" needing to contact Keystone, will, I assume,
consult this catalog and "pick one".

So if I use a DNS round-robin for the two keystones:

openstack.domain.tld.    1 IN A 10.0.4.1
openstack.domain.tld.    1 IN A 10.0.4.9

the entry for 'openstack.domain.tld.' will be invalidated every second,
practically making sure that a new request will be given every time.
The downside is that if the .1 keystone is down, the service will have
to wait for the time out before it can ask the next one.

Round-robin is "poor mans load balancer", and it have many flaws, but
it will at least give some form of using all available resources at any
one time.

For it I use the same virtual IP address for both keystone instances
managed by keepalived.

That is more like a hot-standby (only one server is used at any one time)
and I guess that's an option as well. But because I have limited resources,
I'd prefer, if possible, to use all of them all the time.

Balancing the load is less of an issue than high availability is, but
if possible, I'd like to solve both of them :).

Also you can use peacemaker and other stuff to reach high availability

Yes, but I'm guessing those need another machine in front of the ones
I want to load balance. And if that goes down, EVERYTHING stops working.
Unless they are clustered, which require even more machines!

Which I don't have. I don't want to dedicate a dual CPU, eight core
Intel Xeon E5530 @ 2.40GHz just to swap traffic around! It's a huge
waste of precious resources. And if I add smaller machines outside of
the blade center, then ALL traffic needs to go out in the rack and then
back in, which will affect performance (which is already kind'a bad
because it's an older setup with only Gbps links).

I had to dedicate one whole switch (there's two Cisco 3020 in the
blade center) just for the truncated link down to the storage. Which
is another think I need to solve eventually - it is currently the biggest,
badest SPOF :( :(.
--
Michael Jackson is not going to buried or cremated
but recycled into shopping bags so he can remain white,
plastic and dangerous for kids to play with.

responded Sep 20, 2016 by Turbo_Fredriksson (8,980 points)   7 12 15
0 votes

Also you can use peacemaker and other stuff to reach high availability
Yes, but I'm guessing those need another machine in front of the ones
I want to load balance. And if that goes down, EVERYTHING stops working.
Unless they are clustered, which require even more machines!

If I had these constraints I would add a loadbalancer-config on the same machine that runs the OpenStack apis.
So add keepalived with haproxy on the 2 “OpenStack controllers”.
Keepalived for a VRRP based HA-IP and haproxy for load-balancing (with the nice addition you can do SSL offloading on haproxy).
This is somewhat similar to what what most vendors (Mirantis/Ubuntu/RedHat) are doing AFAIK.

If you use neutron with the L3 agent you will also have to think about HA of that.
That’s a different story though and I am not really up to speed on the latest and greatest there.

Cheers,
Robert


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 21, 2016 by Van_Leeuwen,_Robert (1,740 points)   1 3
...