settingsLogin | Registersettings

[openstack-dev] [nova] Should we add the 'force' option to the cold migrate API too?

0 votes

Given the recent bugs [1][2] due to the force flag in the live migrate
and evacuate APIs related to Placement, and some other long standing
bugs about bypassing the scheduler [3], I don't think we should add the
force option to the cold migrate API, as (re-)proposed in Takashi's cold
migrate spec here [4].

I'm fine with being able to specify a host during cold migrate/resize,
but I think the specified host should be validated by the scheduler (and
placement) so that the instance can actually move to that specified
destination host.

Since we've built more logic into the scheduler in Pike for integration
with Placement, bypassing that gets us into maintenance issues with
having to duplicate code throughout conductor and just in general, seems
like a bad idea to force a host and bypass the scheduler and potentially
break the instance. Not to mention the complicated logic of passing the
host through from the API to conductor to the scheduler is it's own
maintenance problem [5].

Looking back at when the force flag was added to the APIs, it was from
this spec [6]. Reading that, before that microversion if a host was
specified we'd bypass the scheduler, so the force flag was really just
there for backward compatibility I guess in case you wanted the option
to break the instance or your deployment. :) Otherwise after that
microversion if you specify a host but not the force flag, then we
validate the specified host via the scheduler first. Given this, and the
fact we don't have any backward compatibility to maintain with
specifying a host for cold migrate, I don't think we need to add a force
flag for it, unless people really love that option on the live migrate
and evacuate APIs, but it just seems overly dangerous to me.

Finally, if one is going to make the argument, "but this would be
consistent with the live migrate and evacuate APIs", I can also point
out that we don't allow you to specify a host (forced or not) during
unshelve of a shelved offloaded instance - which is basically a move
(new build on a new host chosen by the scheduler). I'm not advocating
that we make unshelve more complicated though, because that's already
broken in several known ways [7][8][9].

[1] https://bugs.launchpad.net/nova/+bug/1712008
[2] https://bugs.launchpad.net/nova/+bug/1713786
[3] https://bugs.launchpad.net/nova/+bug/1427772
[4] https://review.openstack.org/#/c/489031/
[5]
http://lists.openstack.org/pipermail/openstack-dev/2017-August/121342.html
[6]
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/check-destination-on-migrations.html
[7] https://bugs.launchpad.net/nova/+bug/1675791
[8] https://bugs.launchpad.net/nova/+bug/1627694
[9] https://bugs.launchpad.net/nova/+bug/1547142

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 31, 2017 in openstack-dev by mriedemos_at_gmail.c (15,720 points)   2 4 10

4 Responses

0 votes

On 08/30/2017 09:09 AM, Matt Riedemann wrote:
Given the recent bugs [1][2] due to the force flag in the live migrate and
evacuate APIs related to Placement, and some other long standing bugs about
bypassing the scheduler [3], I don't think we should add the force option to the
cold migrate API, as (re-)proposed in Takashi's cold migrate spec here [4].

I'm fine with being able to specify a host during cold migrate/resize, but I
think the specified host should be validated by the scheduler (and placement) so
that the instance can actually move to that specified destination host.

Since we've built more logic into the scheduler in Pike for integration with
Placement, bypassing that gets us into maintenance issues with having to
duplicate code throughout conductor and just in general, seems like a bad idea
to force a host and bypass the scheduler and potentially break the instance. Not
to mention the complicated logic of passing the host through from the API to
conductor to the scheduler is it's own maintenance problem [5].

I completely agree with all of this.

Now that nova properly tracks non-shareable resources over cold migration
(things like hugepages and PCI devices that cannot be shared) it really doesn't
make sense to bypass the scheduler since it could end up seriously confusing the
resource tracking mechanisms.

(We might even want to fail a live migration/evacuation with a forced
destination that could cause a conflict in these non-shareable resources, but
that'd be a behaviour change and therefore a new microversion.)

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

On 8/30/2017 11:35 AM, Chris Friesen wrote:
(We might even want to fail a live migration/evacuation with a forced
destination that could cause a conflict in these non-shareable
resources, but that'd be a behaviour change and therefore a new
microversion.)

That's https://bugs.launchpad.net/nova/+bug/1427772 I believe, and I
don't think we should need a microversion to fix broken behavior in the
backend. As noted, even with a forced host live migration, we still do
some things like the ComputeFilter and RamFilter checks within conductor
itself.

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 10
0 votes

On 08/30/2017 10:56 AM, Matt Riedemann wrote:
On 8/30/2017 11:35 AM, Chris Friesen wrote:

(We might even want to fail a live migration/evacuation with a forced
destination that could cause a conflict in these non-shareable resources, but
that'd be a behaviour change and therefore a new microversion.)

That's https://bugs.launchpad.net/nova/+bug/1427772 I believe, and I don't think
we should need a microversion to fix broken behavior in the backend. As noted,
even with a forced host live migration, we still do some things like the
ComputeFilter and RamFilter checks within conductor itself.

I think you're correct, and that bug seems like it would cover the desired
behaviour.

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

On Wed, Aug 30, 2017 at 5:09 PM, Matt Riedemann mriedemos@gmail.com wrote:

Given the recent bugs [1][2] due to the force flag in the live migrate and
evacuate APIs related to Placement, and some other long standing bugs about
bypassing the scheduler [3], I don't think we should add the force option
to the cold migrate API, as (re-)proposed in Takashi's cold migrate spec
here [4].

I'm fine with being able to specify a host during cold migrate/resize, but
I think the specified host should be validated by the scheduler (and
placement) so that the instance can actually move to that specified
destination host.

Since we've built more logic into the scheduler in Pike for integration
with Placement, bypassing that gets us into maintenance issues with having
to duplicate code throughout conductor and just in general, seems like a
bad idea to force a host and bypass the scheduler and potentially break the
instance. Not to mention the complicated logic of passing the host through
from the API to conductor to the scheduler is it's own maintenance problem
[5].

Looking back at when the force flag was added to the APIs, it was from
this spec [6]. Reading that, before that microversion if a host was
specified we'd bypass the scheduler, so the force flag was really just
there for backward compatibility

Indeed. That said, I've heard some ops wanting to migrate instances to
computes where resources were not possibly enough to accept the instance
but where it's preferred to have performance problems than just stopped
instances.
If you think about the move operations using the force flag (evacuate and
live-migrate), those were used by operators when they had a problem with a
compute node and they wanted to evacuate very quickly instances.

I guess in case you wanted the option to break the instance or your
deployment. :) Otherwise after that microversion if you specify a host but
not the force flag, then we validate the specified host via the scheduler
first. Given this, and the fact we don't have any backward compatibility to
maintain with specifying a host for cold migrate, I don't think we need to
add a force flag for it, unless people really love that option on the live
migrate and evacuate APIs, but it just seems overly dangerous to me.

While I understand operators wanting to evacuate instances (or rebuilding
them by using the evacuation API) in case they see problems with hosts, I
don't see why we should need to have a "force" flag for a cold migration if
you're passing a target.
Say :
- either your compute node is down and then you need to recreate your
customers' instances very quickly : then you call "nova evacuate".
- or your compute node is still alive but you want to migrate quickly
without telling your customers : then you use "nova live-migrate".

I don't see cases where operators (because passing a target requires you to
be an admin) would like to cold migrate instances for their customers
without communicating them a specific timeline for the move operation and
so quickly that it would require to use a force flag to bypass the
scheduler.
Maybe I'm wrong but I'm fine with asking Takashi to not add the force flag
in his implementation for the cold migration API and wait for people
wanting to have that flag to propose a specific specification that would
describe the use-case.

Finally, if one is going to make the argument, "but this would be
consistent with the live migrate and evacuate APIs", I can also point out
that we don't allow you to specify a host (forced or not) during unshelve
of a shelved offloaded instance - which is basically a move (new build on a
new host chosen by the scheduler). I'm not advocating that we make unshelve
more complicated though, because that's already broken in several known
ways [7][8][9].

Well, we don't have consistent APIs anyway. If you think about all the move
operations plus the boot request itself, each of them is already very
different from the other from an API perspective. Yay.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 31, 2017 by Sylvain_Bauza (14,100 points)   1 3 5
...