settingsLogin | Registersettings

[openstack-dev] [Magnum] Scheduling for Magnum

0 votes

Magnum Team,

In our initial spec, we addressed the subject of resource scheduling. Our plan was to begin with a naive scheduler that places resources on a specified Node and can sequentially fill Nodes if one is not specified.

Magnum supports multiple conductor backends[1], one of which is our Kubernetes backend. We also have a native Docker backend that we would like to enhance so that when pods or containers are created, the target nodes can be selected according to user-supplied filters. Some examples of this are:

Constraint, Affinity, Anti-Affinity, Health

We have multiple options for solving this challenge. Here are a few:

1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].
3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner.
4) Write our own filter scheduler, inspired by Nova.

I suggest that we deserve to have a scheduling implementation for our native docker backend before Gantt is ready. It’s unrealistic that the Magnum team will be able to accelerate Gantt’s progress, as significant changes must be made in Nova for this to happen. The Nova team is best equipped to do this. It requires active participation from Nova’s core review team, which has limited bandwidth, and other priorities to focus on. I think we unanimously agree that we would prefer to use Gantt, if it were available sooner.

I suggest we also rule out option 4, because it amounts to re-inventing the wheel.

That leaves us with options 1 and 2 in the short term. The disadvantage of either of these approaches is that we will likely need to remove them and replace them with Gantt (or a derivative work) once it matures. The advantage of option 1 is that python code already exists for this, and we know it works well in Nova. We could cherry pick that over, and drop it directly into Magnum. The advantage of option 2 is that we leverage the talents of the developers working on Swarm, which is better than option 4. New features are likely to surface in parallel with our efforts, so we would enjoy the benefits of those without expending work in our own project.

So, how do you feel about options 1 and 2? Which do you feel is more suitable for Magnum? What other options should we consider that might be better than either of these choices?

I have a slight preference for option 2 - integrating with Swarm, but I could be persuaded to pick option 1, or something even more brilliant. Please discuss.

Thanks,

Adrian

[1] https://github.com/stackforge/magnum/tree/master/magnum/conductor/handlers
[2] https://github.com/docker/swarm/tree/master/scheduler/
[3] https://wiki.openstack.org/wiki/Gantt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Feb 7, 2015 in openstack-dev by Adrian_Otto (11,060 points)   2 4 7
retagged Apr 14, 2015 by admin

20 Responses

0 votes

On Sat, 2015-02-07 at 00:44 +0000, Adrian Otto wrote:
Magnum Team,

In our initial spec, we addressed the subject of resource scheduling. Our plan was to begin with a naive scheduler that places resources on a specified Node and can sequentially fill Nodes if one is not specified.

Magnum supports multiple conductor backends[1], one of which is our Kubernetes backend. We also have a native Docker backend that we would like to enhance so that when pods or containers are created, the target nodes can be selected according to user-supplied filters. Some examples of this are:

Constraint, Affinity, Anti-Affinity, Health

We have multiple options for solving this challenge. Here are a few:

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].
3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova.
This is expected to happen about a year from now, possibly sooner.
4) Write our own filter scheduler, inspired by Nova.

I suggest that we deserve to have a scheduling implementation for our
native docker backend before Gantt is ready. It’s unrealistic that the
Magnum team will be able to accelerate Gantt’s progress, as
significant changes must be made in Nova for this to happen. The Nova
team is best equipped to do this. It requires active participation
from Nova’s core review team, which has limited bandwidth, and other
priorities to focus on. I think we unanimously agree that we would
prefer to use Gantt, if it were available sooner.

I suggest we also rule out option 4, because it amounts to
re-inventing the wheel.

That leaves us with options 1 and 2 in the short term. The
disadvantage of either of these approaches is that we will likely need
to remove them and replace them with Gantt (or a derivative work) once
it matures. The advantage of option 1 is that python code already
exists for this, and we know it works well in Nova. We could cherry
pick that over, and drop it directly into Magnum. The advantage of
option 2 is that we leverage the talents of the developers working on
Swarm, which is better than option 4. New features are likely to
surface in parallel with our efforts, so we would enjoy the benefits
of those without expending work in our own project.

So, how do you feel about options 1 and 2? Which do you feel is more
suitable for Magnum? What other options should we consider that might
be better than either of these choices?

I have a slight preference for option 2 - integrating with Swarm, but
I could be persuaded to pick option 1, or something even more
brilliant. Please discuss.

Got to say that Option 1 looks far preferable. As you say, we have to
switch to Gantt eventually, so it might end up being an expensive and
difficult retro fit with Option 2. With Option 1, we look mostly like
the Nova scheduler, so can let them take the initial hit of doing the
shift to Gantt and slipstream in their wake once the major pain points
are ironed out.

James


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 7, 2015 by James_Bottomley (2,940 points)   2 2
0 votes

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the
Docker API, although they're only about 75% compatible at the moment.
Ideally, the Docker backend would work with both single docker hosts and
clusters of Docker machines powered by Swarm. It would be nice, however, if
scheduler hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 7, 2015 by Eric_Windisch (980 points)   1
0 votes

From: Eric Windisch eric@windisch.us
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve.

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 7, 2015 by Steven_Dake_(stdake) (24,540 points)   2 10 24
0 votes

I second that. my first instinct is the same as Steve.

-- dims

On Sat, Feb 7, 2015 at 1:24 PM, Steven Dake (stdake) stdake@cisco.com wrote:

From: Eric Windisch eric@windisch.us
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the Docker
API, although they're only about 75% compatible at the moment. Ideally, the
Docker backend would work with both single docker hosts and clusters of
Docker machines powered by Swarm. It would be nice, however, if scheduler
hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with swarm
and leave out any cherry-picking from Nova. It would be better to integrate
scheduling hints into Swarm, but I’m sure the swarm upstream is busy with
requests and this may be difficult to achieve.

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 7, 2015 by Davanum_Srinivas (35,920 points)   2 4 8
0 votes

Ok, so if we proceed using Swarm as our first pursuit, and we want to add things to Swarm like scheduling hints, we should open a Magnum bug ticket to track each of the upstream patches, and I can help to bird dog those. We should not shy away from upstream enhancements until we get firm feedback suggesting our contributions are out of scope.

Adrian

-------- Original message --------
From: "Steven Dake (stdake)"
Date:02/07/2015 10:27 AM (GMT-08:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

From: Eric Windisch eric@windisch.us
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve.

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 8, 2015 by Adrian_Otto (11,060 points)   2 4 7
0 votes

Sorry for late. I'm OK with both nova scheduler and swarm as both of them
using same logic for scheduling: filter + strategy (weight), and the code
structure/logic is also very similar between nova scheduler and swarm.

In my understanding, even if we use swarm and translate go to python, after
this scheduler is added to magnum, we may notice that it is very similar
with nova scheduler.

Thanks!

2015-02-08 11:01 GMT+08:00 Adrian Otto adrian.otto@rackspace.com:

Ok, so if we proceed using Swarm as our first pursuit, and we want to add
things to Swarm like scheduling hints, we should open a Magnum bug ticket
to track each of the upstream patches, and I can help to bird dog those. We
should not shy away from upstream enhancements until we get firm feedback
suggesting our contributions are out of scope.

Adrian

-------- Original message --------
From: "Steven Dake (stdake)"
Date:02/07/2015 10:27 AM (GMT-08:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

From: Eric Windisch eric@windisch.us
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the
Docker API, although they're only about 75% compatible at the moment.
Ideally, the Docker backend would work with both single docker hosts and
clusters of Docker machines powered by Swarm. It would be nice, however, if
scheduler hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with
swarm and leave out any cherry-picking from Nova. It would be better to
integrate scheduling hints into Swarm, but I’m sure the swarm upstream is
busy with requests and this may be difficult to achieve.

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks,

Jay Lau (Guangya Liu)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 8, 2015 by Jay_Lau (7,320 points)   1 8 11
0 votes

Hi Magnum team,

Le 07/02/2015 19:24, Steven Dake (stdake) a écrit :

From: Eric Windisch <eric@windisch.us eric@windisch.us>
Reply-To: "OpenStack Development Mailing List (not for usage
questions)" <openstack-dev@lists.openstack.org
openstack-dev@lists.openstack.org>
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

    1) Cherry pick scheduler code from Nova, which already has a
    working a filter scheduler design.

The Gantt team explored that option by the Icehouse cycle and it failed
with a lot of problems. I won't list all of those, but I'll just explain
that we discovered how the Scheduler and the Nova compute manager were
tighly coupled, which was meaning that a repository fork was really
difficult to do without reducing the tech debt.

That said, our concerns were far different from the Magnum team : it was
about having feature parity and replacing the current Nova scheduler,
while your team is just saying that they want to have something about
containers.

    2) Integrate swarmd to leverage its scheduler[2].


I see #2 as not an alternative but possibly an "also". Swarm uses
the Docker API, although they're only about 75% compatible at the
moment. Ideally, the Docker backend would work with both single
docker hosts and clusters of Docker machines powered by Swarm. It
would be nice, however, if scheduler hints could be passed from
Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with
swarm and leave out any cherry-picking from Nova. It would be better
to integrate scheduling hints into Swarm, but I’m sure the swarm
upstream is busy with requests and this may be difficult to achieve.

I don't want to give my opinion about which option you should take as I
don't really know your needs. If I understand correctly, this is about
having a scheduler providing affinity rules for containers. Do you have
a document explaining which interfaces you're looking for, which kind of
APIs you're wanting or what's missing with the current Nova scheduler ?

MHO is that the technology shouldn't drive your decision : whatever the
backend is (swarmd or an inherited nova scheduler), your interfaces
should be the same.

-Sylvain

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 9, 2015 by Sylvain_Bauza (14,100 points)   1 3 5
0 votes

Thanks Sylvain, we did not work out the API requirement till now but I
think the requirement should be similar with nova: we need
select_destination to select the best target host based on filters and
weights.

There are also some discussions here
https://blueprints.launchpad.net/magnum/+spec/magnum-scheduler-for-docker

Thanks!

2015-02-09 16:22 GMT+08:00 Sylvain Bauza sbauza@redhat.com:

Hi Magnum team,

Le 07/02/2015 19:24, Steven Dake (stdake) a écrit :

From: Eric Windisch eric@windisch.us
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
Date: Saturday, February 7, 2015 at 10:09 AM
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.

The Gantt team explored that option by the Icehouse cycle and it failed
with a lot of problems. I won't list all of those, but I'll just explain
that we discovered how the Scheduler and the Nova compute manager were
tighly coupled, which was meaning that a repository fork was really
difficult to do without reducing the tech debt.

That said, our concerns were far different from the Magnum team : it was
about having feature parity and replacing the current Nova scheduler, while
your team is just saying that they want to have something about containers.

2) Integrate swarmd to leverage its scheduler[2].

I see #2 as not an alternative but possibly an "also". Swarm uses the
Docker API, although they're only about 75% compatible at the moment.
Ideally, the Docker backend would work with both single docker hosts and
clusters of Docker machines powered by Swarm. It would be nice, however, if
scheduler hints could be passed from Magnum to Swarm.

Regards,
Eric Windisch

Adrian & Eric,

I would prefer to keep things simple and just integrate directly with
swarm and leave out any cherry-picking from Nova. It would be better to
integrate scheduling hints into Swarm, but I’m sure the swarm upstream is
busy with requests and this may be difficult to achieve.

I don't want to give my opinion about which option you should take as I
don't really know your needs. If I understand correctly, this is about
having a scheduler providing affinity rules for containers. Do you have a
document explaining which interfaces you're looking for, which kind of APIs
you're wanting or what's missing with the current Nova scheduler ?

MHO is that the technology shouldn't drive your decision : whatever the
backend is (swarmd or an inherited nova scheduler), your interfaces should
be the same.

-Sylvain

Regards
-steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks,

Jay Lau (Guangya Liu)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 9, 2015 by Jay_Lau (7,320 points)   1 8 11
0 votes

Adrian Otto wrote:
[...]
We have multiple options for solving this challenge. Here are a few:

1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].
3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner.
4) Write our own filter scheduler, inspired by Nova.

I haven't looked enough into Swarm to answer that question myself, but
how much would #2 tie Magnum to Docker containers ?

There is value for Magnum to support other container engines / formats
(think Rocket/Appc) in the long run, so we should avoid early design
choices that would prevent such support in the future.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 9, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

On 2/9/15, 3:02 AM, "Thierry Carrez" thierry@openstack.org wrote:

Adrian Otto wrote:
[...]
We have multiple options for solving this challenge. Here are a few:

1) Cherry pick scheduler code from Nova, which already has a working a
filter scheduler design.
2) Integrate swarmd to leverage its scheduler[2].
3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova.
This is expected to happen about a year from now, possibly sooner.
4) Write our own filter scheduler, inspired by Nova.

I haven't looked enough into Swarm to answer that question myself, but
how much would #2 tie Magnum to Docker containers ?

There is value for Magnum to support other container engines / formats
(think Rocket/Appc) in the long run, so we should avoid early design
choices that would prevent such support in the future.

Thierry,
Magnum has an object type of a bay which represents the underlying cluster
architecture used. This could be kubernetes, raw docker, swarmd, or some
future invention. This way Magnum can grow independently of the
underlying technology and provide a satisfactory user experience dealing
with the chaos that is the container development world :)

We will absolutely support relevant container technology, likely through
new Bay formats (which are really just heat templates).

Regards
-steve

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 9, 2015 by Steven_Dake_(stdake) (24,540 points)   2 10 24
...