settingsLogin | Registersettings

[Openstack-operators] Fwd: Re: [openstack-dev] [nova][ironic] Concerns over rigid resource class-only ironic scheduling

0 votes

Forgot to copy the ops list on my reply.

-------- Forwarded Message --------
Subject: Re: [openstack-dev] [nova][ironic] Concerns over rigid resource
class-only ironic scheduling
Date: Thu, 7 Sep 2017 14:57:24 -0500
From: Matt Riedemann mriedem.os@gmail.com
To: openstack-dev@lists.openstack.org

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Some more background information is in the ironic spec here:

https://review.openstack.org/#/c/500429/

Also, be aware of these release notes for Pike related to baremetal
scheduling:

http://docs-draft.openstack.org/77/501477/1/check/gate-nova-releasenotes/1dc7513//releasenotes/build/html/unreleased.html#id2

In Pike, nova is using a combination of VCPU/MEMORYMB/DISKGB resource
class amounts from the flavor during scheduling as it always has, but it
will also check for the custom resourceclass which comes from the
ironic node. The custom resource class is optional in Pike but will be a
hard requirement in Queens, or at least that was the plan. The idea
being that long-term we'd stop consulting VCPU/MEMORY
MB/DISKGB from
the flavor during scheduling and just use the atomic node.resource
class
since we want to allocate a nova instance to an entire ironic node, and
this is also why the Exact* filters were used too.

There are more details on using custom resource classes for scheduling here:

https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/custom-resource-classes-in-flavors.html

Nisha is raising the question about whether or not we're making
incorrect assumptions about how people are using nova/ironic and they
want to use the non-Exact filters for VCPU/MEMORYMB/DISKGB, which as
far as I have ever heard is not recommended/supported upstream as it can
lead to resource tracking issues in Nova that eventually lead to
scheduling failures later because of the scheduler thinking a node is
available for more than one instance when it's really not.

--

Thanks,

Matt


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Sep 14, 2017 in openstack-operators by mriedemos_at_gmail.c (15,720 points)   2 4 7

6 Responses

0 votes

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class from
nova. Do you guys see any concerns over this "rigid resource class only
ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 7, 2017 by Nisha_Agarwal (600 points)   1
0 votes

Nisha is raising the question about whether or not we're making incorrect
assumptions >>about how people are using nova/ironic and they want to use
the non-Exact filters for >>VCPU/MEMORYMB/DISKGB, which as far as I have
ever heard is not >>recommended/supported upstream as it can lead to
resource tracking issues in Nova that >>eventually lead to scheduling
failures later because of the scheduler thinking a node is >>available for
more than one instance when it's really not.

Just to clarify, I havent heard about this issue lately when we use
non-Exact filters. (before Pike release).

Regards
Nisha

On Fri, Sep 8, 2017 at 1:27 AM, Matt Riedemann mriedemos@gmail.com wrote:

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Some more background information is in the ironic spec here:

https://review.openstack.org/#/c/500429/

Also, be aware of these release notes for Pike related to baremetal
scheduling:

http://docs-draft.openstack.org/77/501477/1/check/gate-nova-
releasenotes/1dc7513//releasenotes/build/html/unreleased.html#id2

In Pike, nova is using a combination of VCPU/MEMORYMB/DISKGB resource
class amounts from the flavor during scheduling as it always has, but it
will also check for the custom resourceclass which comes from the ironic
node. The custom resource class is optional in Pike but will be a hard
requirement in Queens, or at least that was the plan. The idea being that
long-term we'd stop consulting VCPU/MEMORY
MB/DISKGB from the flavor
during scheduling and just use the atomic node.resource
class since we want
to allocate a nova instance to an entire ironic node, and this is also why
the Exact* filters were used too.

There are more details on using custom resource classes for scheduling
here:

https://specs.openstack.org/openstack/nova-specs/specs/pike/
approved/custom-resource-classes-in-flavors.html

Nisha is raising the question about whether or not we're making incorrect
assumptions about how people are using nova/ironic and they want to use the
non-Exact filters for VCPU/MEMORYMB/DISKGB, which as far as I have ever
heard is not recommended/supported upstream as it can lead to resource
tracking issues in Nova that eventually lead to scheduling failures later
because of the scheduler thinking a node is available for more than one
instance when it's really not.

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
The Secret Of Success is learning how to use pain and pleasure, instead
of having pain and pleasure use you. If You do that you are in control
of your life. If you don't life controls you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 7, 2017 by Nisha_Agarwal (600 points)   1
0 votes

Hello Operators,

It will help to consider this change for Queens release if you could let us
know your opinion/concerns over this change done in Pike release.

Regards
Nisha

On Fri, Sep 8, 2017 at 1:36 AM, Nisha Agarwal agarwalnisha1980@gmail.com
wrote:

Nisha is raising the question about whether or not we're making
incorrect assumptions >>about how people are using nova/ironic and they
want to use the non-Exact filters for >>VCPU/MEMORYMB/DISKGB, which as
far as I have ever heard is not >>recommended/supported upstream as it can
lead to resource tracking issues in Nova that >>eventually lead to
scheduling failures later because of the scheduler thinking a node is
available for more than one instance when it's really not.

Just to clarify, I havent heard about this issue lately when we use
non-Exact filters. (before Pike release).

Regards
Nisha

On Fri, Sep 8, 2017 at 1:27 AM, Matt Riedemann mriedemos@gmail.com
wrote:

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter
in the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Some more background information is in the ironic spec here:

https://review.openstack.org/#/c/500429/

Also, be aware of these release notes for Pike related to baremetal
scheduling:

http://docs-draft.openstack.org/77/501477/1/check/gate-nova-
releasenotes/1dc7513//releasenotes/build/html/unreleased.html#id2

In Pike, nova is using a combination of VCPU/MEMORYMB/DISKGB resource
class amounts from the flavor during scheduling as it always has, but it
will also check for the custom resourceclass which comes from the ironic
node. The custom resource class is optional in Pike but will be a hard
requirement in Queens, or at least that was the plan. The idea being that
long-term we'd stop consulting VCPU/MEMORY
MB/DISKGB from the flavor
during scheduling and just use the atomic node.resource
class since we want
to allocate a nova instance to an entire ironic node, and this is also why
the Exact* filters were used too.

There are more details on using custom resource classes for scheduling
here:

https://specs.openstack.org/openstack/nova-specs/specs/pike/
approved/custom-resource-classes-in-flavors.html

Nisha is raising the question about whether or not we're making incorrect
assumptions about how people are using nova/ironic and they want to use the
non-Exact filters for VCPU/MEMORYMB/DISKGB, which as far as I have ever
heard is not recommended/supported upstream as it can lead to resource
tracking issues in Nova that eventually lead to scheduling failures later
because of the scheduler thinking a node is available for more than one
instance when it's really not.

--

Thanks,

Matt



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
The Secret Of Success is learning how to use pain and pleasure, instead
of having pain and pleasure use you. If You do that you are in control
of your life. If you don't life controls you.

--
The Secret Of Success is learning how to use pain and pleasure, instead
of having pain and pleasure use you. If You do that you are in control
of your life. If you don't life controls you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 11, 2017 by Nisha_Agarwal (600 points)   1
0 votes

On Thu, 7 Sep 2017 14:57:24 -0500, Matt Riedemann wrote:
Some more background information is in the ironic spec here:

https://review.openstack.org/#/c/500429/

Also, be aware of these release notes for Pike related to baremetal
scheduling:

http://docs-draft.openstack.org/77/501477/1/check/gate-nova-releasenotes/1dc7513//releasenotes/build/html/unreleased.html#id2

In Pike, nova is using a combination of VCPU/MEMORYMB/DISKGB resource
class amounts from the flavor during scheduling as it always has, but it
will also check for the custom resourceclass which comes from the
ironic node. The custom resource class is optional in Pike but will be a
hard requirement in Queens, or at least that was the plan. The idea
being that long-term we'd stop consulting VCPU/MEMORY
MB/DISKGB from
the flavor during scheduling and just use the atomic node.resource
class
since we want to allocate a nova instance to an entire ironic node, and
this is also why the Exact* filters were used too.

There are more details on using custom resource classes for scheduling
here:

https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/custom-resource-classes-in-flavors.html

Nisha is raising the question about whether or not we're making
incorrect assumptions about how people are using nova/ironic and they
want to use the non-Exact filters for VCPU/MEMORYMB/DISKGB, which as
far as I have ever heard is not recommended/supported upstream as it can
lead to resource tracking issues in Nova that eventually lead to
scheduling failures later because of the scheduler thinking a node is
available for more than one instance when it's really not.

This came up in the Nova PTG room yesterday and I wanted to reply on the
thread with what I understood about it, for those who weren't in the
session. In general, it's recommended to use the exact filters (1 flavor
per Ironic node hardware config) as there's no concept of partially
claiming a baremetal node.

But, with the old non-exact filters, you could get away with creating
fewer flavors than you have hardware configs and get "fuzzy matching" on
Ironic nodes, to get nodes whose configs are "close enough" but not
exact. This might be helpful in situations where you have some oddball
configs you don't want to have separate flavors for.
I was thinking, if it's possible to assign more than one resource class
to an Ironic node, maybe you could get similar behavior to the old
non-exact filters. So if you have an oddball config, you could tag it as
multiple resource classes that it's "close enough" to for a match. But
I'm not sure whether it's possible for an Ironic node to be tagged with
more than one resource class.

-melanie


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 14, 2017 by melanie_witt (3,180 points)   2 5
0 votes

On Sep 14, 2017, at 10:30 AM, melanie witt melwittt@gmail.com wrote:

I was thinking, if it's possible to assign more than one resource class to an Ironic node, maybe you could get similar behavior to the old non-exact filters. So if you have an oddball config, you could tag it as multiple resource classes that it's "close enough" to for a match. But I'm not sure whether it's possible for an Ironic node to be tagged with more than one resource class.

On the placement side, having an ironic node with two resource classes such as RC1 and RC2, would mean that the ResourceProvider (the ironic node) would have two inventory records: one for RC1, and another for RC2. When a request for a flavor specifying one of these classes is handled, only the one class’s inventory would be consumed. Placement would think that the node still had one of the other resource class available, and would include that if another request for that class is received, which would then fail as the node is already in use.

-- Ed Leafe


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 14, 2017 by Ed_Leafe (11,720 points)   1 3 5
0 votes

On Thu, 14 Sep 2017 11:15:26 -0600, Ed Leafe wrote:
On Sep 14, 2017, at 10:30 AM, melanie witt melwittt@gmail.com wrote:

I was thinking, if it's possible to assign more than one resource class to an Ironic node, maybe you could get similar behavior to the old non-exact filters. So if you have an oddball config, you could tag it as multiple resource classes that it's "close enough" to for a match. But I'm not sure whether it's possible for an Ironic node to be tagged with more than one resource class.

On the placement side, having an ironic node with two resource classes such as RC1 and RC2, would mean that the ResourceProvider (the ironic node) would have two inventory records: one for RC1, and another for RC2. When a request for a flavor specifying one of these classes is handled, only the one class’s inventory would be consumed. Placement would think that the node still had one of the other resource class available, and would include that if another request for that class is received, which would then fail as the node is already in use.

Okay, so it's not possible to have one Ironic node with one inventory
record be classified as two different resource classes. Never mind that
idea then.

Thanks for pointing that out.

-melanie


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 14, 2017 by melanie_witt (3,180 points)   2 5
...