settingsLogin | Registersettings

[openstack-dev] [kolla] Multi-Regions Support

0 votes

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3]. Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon. This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend. At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL |
|--------+-----------+-----------+-----------+------------------------------|
| AR | keystone | identity | public | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 |

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jan 5, 2017 in openstack-dev by Ronan-Alexandre_Cher (260 points)   1 1
retagged Jan 26, 2017 by admin

9 Responses

0 votes

Thanks Ronan,

Great approach.

I am always expecting multi-region support in Kolla. I hope what you did
can be merged into Kolla.

On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau <
ronan-alexandre.cherrueau@inria.fr> wrote:

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3]. Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon. This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend. At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL
|
|--------+-----------+-----------+-----------+--------------
----------------|
| AR | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5
cf974216ae/ansible/roles/keystone/tasks/register.yml
[5] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
40be75c3a2237adfd1a05178f6f60006R1
[6] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
607e6e5925d0031dafa79eb80d198640R215
[7] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
6cbb63bb9c4843bcf8db4710e588475bR188
[8] https://github.com/BeyondTheClouds/kolla/tree/
7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
ba10dd4575f65e03d50d586febdccbadR72
[10] https://github.com/BeyondTheClouds/kolla/blob/
7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11] https://github.com/openstack/kolla/blob/
11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/
templates/nova.conf.j2#L155

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 5, 2017 by Lei_Zhang (6,360 points)   1 2 6
0 votes

Great stuff! Thank you Ronan. I'd love to see this guide refactored
and submitted to our docs (also take a longer look how to make full
fledged support in kolla tree). Looking for volunteers:)

On 5 January 2017 at 07:59, Jeffrey Zhang zhang.lei.fly@gmail.com wrote:
Thanks Ronan,

Great approach.

I am always expecting multi-region support in Kolla. I hope what you did can
be merged into Kolla.

On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau
ronan-alexandre.cherrueau@inria.fr wrote:

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3]. Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon. This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend. At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL
|

|--------+-----------+-----------+-----------+------------------------------|
| AR | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8]
https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10]
https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 5, 2017 by Michał_Jastrzębski (9,220 points)   1 5 6
0 votes

On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau <
ronan-alexandre.cherrueau@inria.fr> wrote:

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3]. Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon. This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend. At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL
|
|--------+-----------+-----------+-----------+--------------
----------------|
| AR | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

The original value is this:
authuri = {{ internalprotocol }}://{{ kollainternalfqdn }}:{{
keystonepublicport }}

Kolla does SSL via haproxy, not directly at the service itself. So internal
traffic is http, external is https. In this case, 'kollainternalfqdn'
points to the haproxy vip. Its not legacy code, but I am sure it can be
changed to work multiregion in the way you are attempting to do it.

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5
cf974216ae/ansible/roles/keystone/tasks/register.yml
[5] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
40be75c3a2237adfd1a05178f6f60006R1
[6] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
607e6e5925d0031dafa79eb80d198640R215
[7] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
6cbb63bb9c4843bcf8db4710e588475bR188
[8] https://github.com/BeyondTheClouds/kolla/tree/
7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9] https://github.com/BeyondTheClouds/kolla/commit/
7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
ba10dd4575f65e03d50d586febdccbadR72
[10] https://github.com/BeyondTheClouds/kolla/blob/
7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11] https://github.com/openstack/kolla/blob/
11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/
templates/nova.conf.j2#L155

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

None of these pieces seem to be in major conflict with anything else, so it
would just be a matter of upstreaming your work (and tweaking things as we
find them). It looks great!

Thanks,
SamYaple


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 5, 2017 by Sam_Yaple (3,560 points)   3 3
0 votes

Hello,

Thanks for your feedback.

The original value is this:
authuri = {{ internalprotocol }}://{{ kollainternalfqdn }}:{{ keystonepublicport }}

Kolla does SSL via haproxy, not directly at the service itself. So internal traffic is http, external is https. In this case, 'kollainternalfqdn' points to the haproxy vip.

Got it, but if you look at the default definition of
keystone_internal_url', insidegroup_vars/all.yml', you have:

332 keystoneinternalurl: "{{ internalprotocol }}://{{
kolla
internalfqdn }}:{{ keystonepublic_port }}/v3"

As you can see, the definition is close to the original value of
auth_uri' (except the v3). So, it seems to be a better idea to setauthuri' to `keystoneinternal_url' instead of the canonical form.

Especially in the context of multi-regions. In this context, you cannot
change the value of {{ kolla_internal_fqdn }}' because, as you say, it has to target the region's HAProxy vip. However, you have to set the value ofauth_uri' to target the Administrative Region HAProxy vip. So,
by defining in nova/neutron/glance conf:

[keystoneauthtoken]
auth
uri = {{ keystoneinternalurl }}
authurl = {{ keystoneadmin_url }}

Then you can target the Administrative Region HAProxy vip by redefining
keystone_*_url' without redefiningkollainternalfqdn'.

Its not legacy code, but I am sure it can be changed to work multiregion in the way you are attempting to do it.

I was talking about legacy code because, in the stable/newton branch,
the auth_uri' is already set to{{ keystoneinternalurl }}', but
solely for Kubernetes engine[1].

Shall we proceed by pushing to Gerrit with a dedicated page in the
documentation? Also, our patch is based on stable/newton, is it better
if we follow the kolla-ansible master?

[1] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L151

On Thu, Jan 5, 2017 at 7:01 PM, Sam Yaple samuel@yaple.net wrote:
On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau
ronan-alexandre.cherrueau@inria.fr wrote:

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3]. Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon. This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend. At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL
|

|--------+-----------+-----------+-----------+------------------------------|
| AR | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

The original value is this:
authuri = {{ internalprotocol }}://{{ kollainternalfqdn }}:{{
keystonepublicport }}

Kolla does SSL via haproxy, not directly at the service itself. So internal
traffic is http, external is https. In this case, 'kollainternalfqdn'
points to the haproxy vip. Its not legacy code, but I am sure it can be
changed to work multiregion in the way you are attempting to do it.

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8]
https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10]
https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

None of these pieces seem to be in major conflict with anything else, so it
would just be a matter of upstreaming your work (and tweaking things as we
find them). It looks great!

Thanks,
SamYaple


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Ronan-A. Cherrueau


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Ronan-Alexandre_Cher (260 points)   1 1
0 votes

On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote:
Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3].

I don't see an "Admin Region" as part of the OpenStack documentation for
multi-region deployment. I also see LDAP mentioned as the recommended
authentication/IdM store.

Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon.

That's not a region. Those should be shared resources across regions.

This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend.

I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL
system REDIS"? But here, you're using MariaDB -- a non-distributed
database -- for the Keystone component which is the very thing that is
the most highly distributed of all state storage in OpenStack.

So, you are replacing the Nova DB (which doesn't need to be distributed
at all, since it's a centralized control plane piece) within the regions
with a "distributed" NoSQL store (and throwing away transactional safety
I might add) but you're going with a non-distributed traditional RDBMS
for the very piece that needs to be shared, distributed, and
highly-available across OpenStack. I don't understand that.

At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

Again, that's not a region. Those are merely shared services between
regions.

OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ

  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL |
|--------+-----------+-----------+-----------+------------------------------|
| AR | keystone | identity | public | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 |

There shouldn't be an AR region. If the Keystone authentication domain
is indeed shared between OpenStack regions, then an administrative user
should be able to hit any Keystone endpoint in any OpenStack region and
add users/projects/roles, etc. to the shared Keystone data store (or if
using LDAP, the admin should be able to add a user to
ActiveDirectory/ApacheDS in any OpenStack region and have that user
information immediately show up in any of the other regions).

Best,
-jay

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Jay_Pipes (59,760 points)   3 11 14
0 votes

On Fri, Jan 6, 2017 at 8:01 PM, Jay Pipes jaypipes@gmail.com wrote:

On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote:

Hello,

TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.

At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.

First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3].

I don't see an "Admin Region" as part of the OpenStack documentation for
multi-region deployment. I also see LDAP mentioned as the recommended
authentication/IdM store.

Concretely, the deployment

considers /one/ Administrative Region (AR) that contains Keystone and
Horizon.

That's not a region. Those should be shared resources across regions.

This is a Kolla-based deployment, so Keystone is hidden

behind an HAProxy, and has MariaDB and memcached as backend.

I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL
system REDIS"? But here, you're using MariaDB -- a non-distributed database
-- for the Keystone component which is the very thing that is the most
highly distributed of all state storage in OpenStack.

This should be read as MariaDB+Galera for replication. It is a
highly-available database.

So, you are replacing the Nova DB (which doesn't need to be distributed at
all, since it's a centralized control plane piece) within the regions with
a "distributed" NoSQL store (and throwing away transactional safety I might
add) but you're going with a non-distributed traditional RDBMS for the very
piece that needs to be shared, distributed, and highly-available across
OpenStack. I don't understand that.

At the

same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:

Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached

Again, that's not a region. Those are merely shared services between
regions.

OpenStack Region x (OSRx):

  • control:

    • HAProxy
    • nova-api/conductor/scheduler
    • neutron-server/l3/dhcp/...
    • glance-api/registry
    • MariaDB
    • RabbitMQ
  • compute1:

    • nova-compute
    • neutron-agent
  • compute2: ...

We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstackregionname'
variable to the name of the current region.

In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:

$ openstack endpoint list

| Region | Serv Name | Serv Type | Interface | URL
|
|--------+-----------+-----------+-----------+-------------
-----------------|
| AR | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal |
http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin |
http://10.24.63.248:35357/v3 |

There shouldn't be an AR region. If the Keystone authentication domain is
indeed shared between OpenStack regions, then an administrative user should
be able to hit any Keystone endpoint in any OpenStack region and add
users/projects/roles, etc. to the shared Keystone data store (or if using
LDAP, the admin should be able to add a user to ActiveDirectory/ApacheDS in
any OpenStack region and have that user information immediately show up in
any of the other regions).

Jay I also picked up on a bit of terminology misuse. There were a few
other bits that were not great. I can't speak for everyone else, but I do
share your concerns about some of this, but I was going to save those
comments for gerrit reviews once the code was rebased to master. I feel
confident this can all work for proper Multi-Region support, though not
as-is.

Thanks,
SamYaple

Best,
-jay

This requires patching the keystone/tasks/register.yml' play[4] to re-execute theCreating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute theCreating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.

In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the node_custome_config' variable to themulti-regions' directory[8] and benefits from Kolla merging config
system.

Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kollakeystoneservice.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.

73 for endpoint in cloud.keystoneclient.endpoints.list():
74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone
client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystoneclient.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint
region)

At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.

74 if endpoint.serviceid == service.id and \
75 endpoint.interface == interface and \
76 _endpoint.region == endpoint
region:

Finally, when we deployed OSRs, we override the value of
keystone_admin_url',keystoneinternalurl' and keystone_public_url' to target Keystone in AR. We also have to change the[keystoneauthtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use thekolla
internalfqdn'
variable instead of `keystone
*_url'?

That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.

So, thanks for the great job Kolla team! And waiting for your
feedback.

[1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936
b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac44
1505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936
b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936
b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936
b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7
cc3136dfc082935e2995a65554/multi-regions
[9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936
b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7
cc3136dfc082935e2995a65554/multi-regions/global.conf
[11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac44
1505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Sam_Yaple (3,560 points)   3 3
0 votes

On 01/06/2017 03:23 PM, Sam Yaple wrote:
This should be read as MariaDB+Galera for replication. It is a
highly-available database.

Don't get me wrong. I love me some Galera. :) However, what the poster
is really working towards is an implementation of the VCPE and eVCPE use
cases for ETSI NFV. These use cases require a highly distributed compute
fabric that can withstand long disruptions in network connectivity
(between POPs/COs and the last mile of network service) while still
being able to service compute and network functions at the customer premise.

Galera doesn't tolerate network disruption of any significant length of
time. At all. If there is a Keystone services running on the customer
premise that is connecting to a Galera database, and that Galera
database's connectivity to its peers is disrupted, down goes the whole
on-premise cloud fabric. And that's exactly what I believe the original
poster is attempting to avoid. Thus my not understanding the choice here.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Jay_Pipes (59,760 points)   3 11 14
0 votes

I was kind of hoping k2k federation would solve that.

one keystone per region to provide a local keystone to talk to,
and a centeral keystone users authenticate with.

Just waiting for horizon to gain support before trying though.

Thanks,
Kevin


From: Jay Pipes [jaypipes@gmail.com]
Sent: Friday, January 06, 2017 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] Multi-Regions Support

On 01/06/2017 03:23 PM, Sam Yaple wrote:
This should be read as MariaDB+Galera for replication. It is a
highly-available database.

Don't get me wrong. I love me some Galera. :) However, what the poster
is really working towards is an implementation of the VCPE and eVCPE use
cases for ETSI NFV. These use cases require a highly distributed compute
fabric that can withstand long disruptions in network connectivity
(between POPs/COs and the last mile of network service) while still
being able to service compute and network functions at the customer premise.

Galera doesn't tolerate network disruption of any significant length of
time. At all. If there is a Keystone services running on the customer
premise that is connecting to a Galera database, and that Galera
database's connectivity to its peers is disrupted, down goes the whole
on-premise cloud fabric. And that's exactly what I believe the original
poster is attempting to avoid. Thus my not understanding the choice here.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

----- Mail original -----

De: "Jay Pipes" jaypipes@gmail.com
À: openstack-dev@lists.openstack.org
Envoyé: Vendredi 6 Janvier 2017 21:42:46
Objet: Re: [openstack-dev] [kolla] Multi-Regions Support

On 01/06/2017 03:23 PM, Sam Yaple wrote:

This should be read as MariaDB+Galera for replication. It is a
highly-available database.

Don't get me wrong. I love me some Galera. :) However, what the poster
is really working towards is an implementation of the VCPE and eVCPE use
cases for ETSI NFV. These use cases require a highly distributed compute
fabric that can withstand long disruptions in network connectivity
(between POPs/COs and the last mile of network service) while still
being able to service compute and network functions at the customer premise.

Galera doesn't tolerate network disruption of any significant length of
time. At all. If there is a Keystone services running on the customer
premise that is connecting to a Galera database, and that Galera
database's connectivity to its peers is disrupted, down goes the whole
on-premise cloud fabric. And that's exactly what I believe the original
poster is attempting to avoid. Thus my not understanding the choice here.

Jay, you are thinking too far ;)

The goal of this thread is to see how Kolla can deploy a multi region scenario.
Remarks/contributions to progress in that direction is the goal of the initial post.

Best,

Matt

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jan 6, 2017 by Matthieu_Simonin (760 points)   1
...