settingsLogin | Registersettings

[openstack-dev] [magnum] Discovery

0 votes

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 16, 2015 in openstack-dev by Daneyon_Hansen_(dane (1,500 points)   2 7

6 Responses

0 votes

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's possible to take it a step further. Could we run etcd in each Bay without using the public discovery endpoint? And then, configure Swarm to simply use the internal ectd as it's discovery mechanism? This could cut one of our external service dependencies and make it easier to run Magnum is an environment with locked down public internet access.?

Anyways, I think #2 could be a good start that we could iterate on later if need be.

--Andrew


From: Daneyon Hansen (danehans) danehans@cisco.com
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm's discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url's to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 17, 2015 by Andrew_Melton (680 points)   1 3
0 votes

+1 for stop using public discovery endpoint, most private cloud vms doesn’t have access to internet and operator must to run etcd instance somewhere just for discovery.


Egor

From: Andrew Melton andrew.melton@RACKSPACE.COM
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Thursday, September 17, 2015 at 12:06
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [magnum] Discovery

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's possible to take it a step further. Could we run etcd in each Bay without using the public discovery endpoint? And then, configure Swarm to simply use the internal ectd as it's discovery mechanism? This could cut one of our external service dependencies and make it easier to run Magnum is an environment with locked down public internet access.​

Anyways, I think #2 could be a good start that we could iterate on later if need be.

--Andrew


From: Daneyon Hansen (danehans) danehans@cisco.com
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 17, 2015 by Egor_Guz (1,080 points)  
0 votes

From: Andrew Melton andrew.melton@RACKSPACE.COM
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Thursday, September 17, 2015 at 12:06 PM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [magnum] Discovery

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's possible to take it a step further. Could we run etcd in each Bay without using the public discovery endpoint? And then, configure Swarm to simply use the internal ectd as it's discovery mechanism? This could cut one of our external service dependencies and make it easier to run Magnum is an environment with locked down public internet access.​

Thanks for your feedback. #2 was the preferred direction of the Magnum Networking Subteam as well. Therefore, I have started working on [1] to move this option forward. As part of this effort, I am slightly refactoring the Swarm heat templates to more closely align them with the k8s templates. Until tcammann completes the larger template refactor [2], I think this will help dev’s more easily implement features across all bay types, distro’s, etc..

We can have etcd and Swarm use a local discovery backend. I have filed bp [3] to establish this effort. As a first step towards [3], I will modify Swarm to use etcd for discovery. However, etcd will still use public discovery for [1]. Either myself or someone from the community will need to attack etcd local discovery as a follow-on. For the longer-term, we may want to consider exposing a --discovery-backend attribute and optionally pass labels to allow users to modify default configurations of the —discovery-backend.

[1] https://review.openstack.org/#/c/224367/
[2] https://blueprints.launchpad.net/magnum/+spec/generate-heat-templates
[3] https://blueprints.launchpad.net/magnum/+spec/bay-type-discovery-options

Anyways, I think #2 could be a good start that we could iterate on later if need be.

--Andrew


From: Daneyon Hansen (danehans) danehans@cisco.com
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 17, 2015 by Daneyon_Hansen_(dane (1,500 points)   2 7
0 votes

In the case where a private cloud is used without access to the Internet, you do have the option of running your own etcd, and configuring that to be used instead.

Adding etcd to every bay should be optional, as a subsequent feature, but should be controlled by a flag in the Baymodel that defaults to off so the public discovery service is used. It might be nice to be able to configure Magnum in an isolated mode which would change the system level default for that flag from off to on.

Maybe the Baymodel resource attribute should be named localdiscoveryservice.

Should turning this on also set the minimum node count for the bay to 3? If not, etcd will not be highly available.

Adrian

On Sep 17, 2015, at 1:01 PM, Egor Guz EGuz@walmartlabs.com wrote:

+1 for stop using public discovery endpoint, most private cloud vms doesn’t have access to internet and operator must to run etcd instance somewhere just for discovery.


Egor

From: Andrew Melton andrew.melton@RACKSPACE.COM
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Thursday, September 17, 2015 at 12:06
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [magnum] Discovery

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's possible to take it a step further. Could we run etcd in each Bay without using the public discovery endpoint? And then, configure Swarm to simply use the internal ectd as it's discovery mechanism? This could cut one of our external service dependencies and make it easier to run Magnum is an environment with locked down public internet access.​

Anyways, I think #2 could be a good start that we could iterate on later if need be.

--Andrew


From: Daneyon Hansen (danehans) danehans@cisco.com
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by Adrian_Otto (11,060 points)   2 4 8
0 votes

Swarm already supports etcd as a discovery backend [1]. So we can implement
both hosted discovery with Docker Hub and using name etcd. And make hosted
discovery with Docker Hub default if discovery_url is not given.

If we run etcd in bay, etcd alse need discovery [2]. Operator should set up
a etcd cluster for other etcd clusters to discover or use public discovery
service. I think it is not necessary to run etcd in swarm cluster just for
discovery service. In a private cloud, operator should set up a local etcd
cluster for discovery service, and all the bays can use it.

[1] https://docs.docker.com/swarm/discovery/
[2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md

Regards,
Wanghua

On Fri, Sep 18, 2015 at 11:39 AM, Adrian Otto adrian.otto@rackspace.com
wrote:

In the case where a private cloud is used without access to the Internet,
you do have the option of running your own etcd, and configuring that to be
used instead.

Adding etcd to every bay should be optional, as a subsequent feature, but
should be controlled by a flag in the Baymodel that defaults to off so the
public discovery service is used. It might be nice to be able to configure
Magnum in an isolated mode which would change the system level default for
that flag from off to on.

Maybe the Baymodel resource attribute should be named
localdiscoveryservice.

Should turning this on also set the minimum node count for the bay to 3?
If not, etcd will not be highly available.

Adrian

On Sep 17, 2015, at 1:01 PM, Egor Guz EGuz@walmartlabs.com wrote:

+1 for stop using public discovery endpoint, most private cloud vms
doesn’t have access to internet and operator must to run etcd instance
somewhere just for discovery.


Egor

From: Andrew Melton <andrew.melton@RACKSPACE.COM>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Date: Thursday, September 17, 2015 at 12:06
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
>
Subject: Re: [openstack-dev] [magnum] Discovery

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's
possible to take it a step further. Could we run etcd in each Bay without
using the public discovery endpoint? And then, configure Swarm to simply
use the internal ectd as it's discovery mechanism? This could cut one of
our external service dependencies and make it easier to run Magnum is an
environment with locked down public internet access.​

Anyways, I think #2 could be a good start that we could iterate on later
if need be.

--Andrew


From: Daneyon Hansen (danehans) <danehans@cisco.com>
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come
across an issue that requires feedback from the community. Here is the
breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting
    this requirement is simple for the kubernetes bay types since kubernetes
    requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum
    implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type.
    Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by
    swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is
    associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example
    etcd
    discoveryurl and swarmdiscovery_url. However, this option would
    needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean
    discovery is similar for both bay types. With both bay types using the same
    mechanism for discovery, it will be easier to provide a private discovery
    option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This
    would require adding support for a different driver that supports
    multi-host networking such as libnetwork. Note: libnetwork is only
    implemented in the Docker experimental release:
    https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share
your thoughts. I would like to obtain feedback from the community before
proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2]
https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 21, 2015 by 王华 (2,080 points)   3
0 votes

All,

Thanks for the feedback and additional ideas related to discovery. For clarity purposes, I would like to circle back to the specific issue that I am experiencing with implementing Flannel for Swarm. Flannel can not be implemented in Swarm bay types without making changes to discovery for Swarm. This is because:

  1. Flannel requires etcd, which is not implemented in Magnum’s Swarm bay type.
  2. The discovery_url is implemented differently among Kubernetes and Swarm bay types, making it impossible for Swarm and etcd discovery to coexist within the same bay type.

I am in the process of moving forward with option 2 of my original email so flannel can be implemented in swarm bay types [1]. I have created a bp [2] to address discovery more holistically. It would be helpful if you could provide your ideas in the whiteboard of the bp.

[1] https://review.openstack.org/#/c/224367/
[2] https://blueprints.launchpad.net/magnum/+spec/bay-type-discovery-options

Regards,
Daneyon Hansen
Software Engineer
Email: danehans@cisco.com
Phone: 303-718-0400
http://about.me/daneyon_hansen

From: 王华 wanghua.humble@gmail.com
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Monday, September 21, 2015 at 1:18 AM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [magnum] Discovery

Swarm already supports etcd as a discovery backend [1]. So we can implement both hosted discovery with Docker Hub and using name etcd. And make hosted discovery with Docker Hub default if discovery_url is not given.

If we run etcd in bay, etcd alse need discovery [2]. Operator should set up a etcd cluster for other etcd clusters to discover or use public discovery service. I think it is not necessary to run etcd in swarm cluster just for discovery service. In a private cloud, operator should set up a local etcd cluster for discovery service, and all the bays can use it.

[1] https://docs.docker.com/swarm/discovery/
[2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md

Regards,
Wanghua

On Fri, Sep 18, 2015 at 11:39 AM, Adrian Otto adrian.otto@rackspace.com wrote:
In the case where a private cloud is used without access to the Internet, you do have the option of running your own etcd, and configuring that to be used instead.

Adding etcd to every bay should be optional, as a subsequent feature, but should be controlled by a flag in the Baymodel that defaults to off so the public discovery service is used. It might be nice to be able to configure Magnum in an isolated mode which would change the system level default for that flag from off to on.

Maybe the Baymodel resource attribute should be named localdiscoveryservice.

Should turning this on also set the minimum node count for the bay to 3? If not, etcd will not be highly available.

Adrian

On Sep 17, 2015, at 1:01 PM, Egor Guz EGuz@walmartlabs.com wrote:

+1 for stop using public discovery endpoint, most private cloud vms doesn’t have access to internet and operator must to run etcd instance somewhere just for discovery.


Egor

From: Andrew Melton <andrew.melton@RACKSPACE.COMandrew.melton@RACKSPACE.COM>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.orgopenstack-dev@lists.openstack.org>
Date: Thursday, September 17, 2015 at 12:06
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.orgopenstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [magnum] Discovery

Hey Daneyon,

I'm fairly partial towards #2 as well. Though, I'm wondering if it's possible to take it a step further. Could we run etcd in each Bay without using the public discovery endpoint? And then, configure Swarm to simply use the internal ectd as it's discovery mechanism? This could cut one of our external service dependencies and make it easier to run Magnum is an environment with locked down public internet access.​

Anyways, I think #2 could be a good start that we could iterate on later if need be.

--Andrew


From: Daneyon Hansen (danehans) <danehans@cisco.comdanehans@cisco.com>
Sent: Wednesday, September 16, 2015 11:26 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [magnum] Discovery

All,

While implementing the flannel --network-driver for swarm, I have come across an issue that requires feedback from the community. Here is the breakdown of the issue:

  1. Flannel [1] requires etcd to store network configuration. Meeting this requirement is simple for the kubernetes bay types since kubernetes requires etcd.
  2. A discovery process is needed for bootstrapping etcd. Magnum implements the public discovery option [2].
  3. A discovery process is also required to bootstrap a swarm bay type. Again, Magnum implements a publicly hosted (Docker Hub) option [3].
  4. Magnum API exposes the discovery_url attribute that is leveraged by swarm and etcd discovery.
  5. Etcd can not be implemented in swarm because discovery_url is associated to swarm’s discovery process and not etcd.

Here are a few options on how to overcome this obstacle:

  1. Make the discoveryurl more specific, for example etcddiscoveryurl and swarmdiscovery_url. However, this option would needlessly expose both discovery url’s to all bay types.
  2. Swarm supports etcd as a discovery backend. This would mean discovery is similar for both bay types. With both bay types using the same mechanism for discovery, it will be easier to provide a private discovery option in the future.
  3. Do not support flannel as a network-driver for k8s bay types. This would require adding support for a different driver that supports multi-host networking such as libnetwork. Note: libnetwork is only implemented in the Docker experimental release: https://github.com/docker/docker/tree/master/experimental.

I lean towards #2 but their may be other options, so feel free to share your thoughts. I would like to obtain feedback from the community before proceeding in a particular direction.

[1] https://github.com/coreos/flannel
[2] https://github.com/coreos/etcd/blob/master/Documentation/discovery_protocol.md
[3] https://docs.docker.com/swarm/discovery/

Regards,
Daneyon Hansen


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 21, 2015 by Daneyon_Hansen_(dane (1,500 points)   2 7
...