settingsLogin | Registersettings

Re: [openstack-dev] [nova][ironic] Concerns over rigid resource class-only ironic scheduling

0 votes

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Some more background information is in the ironic spec here:

https://review.openstack.org/#/c/500429/

Also, be aware of these release notes for Pike related to baremetal
scheduling:

http://docs-draft.openstack.org/77/501477/1/check/gate-nova-releasenotes/1dc7513//releasenotes/build/html/unreleased.html#id2

In Pike, nova is using a combination of VCPU/MEMORYMB/DISKGB resource
class amounts from the flavor during scheduling as it always has, but it
will also check for the custom resourceclass which comes from the
ironic node. The custom resource class is optional in Pike but will be a
hard requirement in Queens, or at least that was the plan. The idea
being that long-term we'd stop consulting VCPU/MEMORY
MB/DISKGB from
the flavor during scheduling and just use the atomic node.resource
class
since we want to allocate a nova instance to an entire ironic node, and
this is also why the Exact* filters were used too.

There are more details on using custom resource classes for scheduling here:

https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/custom-resource-classes-in-flavors.html

Nisha is raising the question about whether or not we're making
incorrect assumptions about how people are using nova/ironic and they
want to use the non-Exact filters for VCPU/MEMORYMB/DISKGB, which as
far as I have ever heard is not recommended/supported upstream as it can
lead to resource tracking issues in Nova that eventually lead to
scheduling failures later because of the scheduler thinking a node is
available for more than one instance when it's really not.

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 19, 2017 in openstack-dev by mriedemos_at_gmail.c (15,720 points)   2 4 5

8 Responses

0 votes
  • Nisha is raising the question about whether or not we're making
    >> incorrect assumptions about how people are using nova/ironic and they
    >> want to use the non-Exact filters for VCPU/MEMORYMB/DISKGB, which as
    >> far as I have ever heard is not recommended/supported upstream as it can
    >> lead to resource tracking issues in Nova that eventually lead to
    >> scheduling failures later because of the scheduler thinking a node is
    >> available for more than one instance when it's really not.

  • This came up in the Nova PTG room yesterday and I wanted to reply on the
    thread with what I understood about it, for those who weren't in the
    session. In general, it's recommended to use the exact filters (1 flavor
    per Ironic node hardware config) as there's no concept of partially
    claiming a baremetal node.

But, with the old non-exact filters, you could get away with creating
fewer flavors than you have hardware configs and get "fuzzy matching" on
Ironic nodes, to get nodes whose configs are "close enough" but not
exact. This might be helpful in situations where you have some oddball
configs you don't want to have separate flavors for.
I was thinking, if it's possible to assign more than one resource class
to an Ironic node, maybe you could get similar behavior to the old
non-exact filters. So if you have an oddball config, you could tag it as
multiple resource classes that it's "close enough" to for a match. But
I'm not sure whether it's possible for an Ironic node to be tagged with
more than one resource class.

  • Nisha is raising the question about whether or not we're making
    > incorrect assumptions about how people are using nova/ironic and they
    > want to use the non-Exact filters for VCPU/MEMORYMB/DISKGB, which as
    > far as I have ever heard is not recommended/supported upstream as it can
    > lead to resource tracking issues in Nova that eventually lead to
    > scheduling failures later because of the scheduler thinking a node is
    > available for more than one instance when it's really not.

  • The concern I have with one single custom resource class for an
    Ironic node is that it takes away some of the options that were
    available before, such as scheduling based on resource quantity and
    non-exact match filters (RamFilter, DiskFilter, and CoreFilter). Nova
    scheduling becomes too restrictive for Ironic.

I know some users are using these options before Pike with no
issues. Therefore, it's a mystery to me whether the non-exact filter
for Ironic really has issues. Even if it has problems, it seems to me
there are ways to address the problem. For instance, Ironic virt
driver can report a node is not available if it's in active state (if
it hasn't done so), or report all resources are consumed when a node
is claimed. Alternatively, Nova scheduler can report all resources
are consumed for an Ironic node if Nova is willing to make such
change. Thanks!

On Thu, Sep 7, 2017 at 12:48 PM, Nisha Agarwal agarwalnisha1980@gmail.com
wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 14, 2017 by Wan-yen_Hsu (480 points)   1
0 votes

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:
Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Bringing this back up.

Does this nova spec from John Garbutt help you?

https://review.openstack.org/#/c/507052/

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 5, 2017 by mriedemos_at_gmail.c (15,720 points)   2 4 5
0 votes

Hi Matt,

As i understand John's spec https://review.openstack.org/#/c/507052/, it is
actually a replacement for capabilities(qualitative only) for ironic. It
doesnt cover the quantitative capabilities as 'traits' are meant only for
qualitative capabilities. Quantitative capabilities are covered by resource
classes in Nova. We have few (one or two) quantitative capabilities already
supported in ironic.

Regards
Nisha

On Thu, Oct 5, 2017 at 9:09 PM, Matt Riedemann mriedemos@gmail.com wrote:

On 9/7/2017 2:48 PM, Nisha Agarwal wrote:

Hi Ironic Operators,

From Pike, ironic nodes get scheduled based on just the resource class
from nova. Do you guys see any concerns over this "rigid resource class
only ironic scheduling"?

To be more specific, at your datacentre/production environment what all
filters are configured in nova.conf (configuration file for nova) for
scheduling an ironic node? Do you use RamFilter/DiskFilter/CoreFilter in
the "usebaremetalfilters" for ironic nodes scheduling from nova?

Thanks and Regards
Nisha



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Bringing this back up.

Does this nova spec from John Garbutt help you?

https://review.openstack.org/#/c/507052/

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
The Secret Of Success is learning how to use pain and pleasure, instead
of having pain and pleasure use you. If You do that you are in control
of your life. If you don't life controls you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 16, 2017 by Nisha_Agarwal (600 points)   1
0 votes

On 10/16/2017 05:31 AM, Nisha Agarwal wrote:
Hi Matt,

As i understand John's spec https://review.openstack.org/#/c/507052/
https://review.openstack.org/#/c/507052/, it is actually a replacement
for capabilities(qualitative only) for ironic. It doesnt cover the
quantitative capabilities as 'traits' are meant only for qualitative
capabilities. Quantitative capabilities are covered by resource classes
in Nova. We have few (one or two) quantitative capabilities already
supported in ironic.

Hi Nisha,

This may be a case of mixed terminology. We do not refer to anything
quantitative as a "capability". Rather, we use the term "resource class"
(or sometimes just "resource") to represent quantitative things that may
be consumed by the instance.

Traits, on the other hand, are qualitative. They represent a binary
on/off capability that the compute host (or baremetal node in the case
of Ironic) exposes.

There's no limit on the number of traits that may be associated with a
particular Ironic baremetal node. However, for Ironic baremetal nodes,
if the node.resourceclass attribute is set, the Nova Ironic virt driver
will create a single inventory record for the node containing a quantity
of 1 and a resource class equal to whatever is in the
node.resource
class attribute. This resource class is auto-created by
Nova as a custom resource class.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by Jay_Pipes (59,760 points)   3 10 13
0 votes

On 19 October 2017 at 15:38, Jay Pipes jaypipes@gmail.com wrote:

On 10/16/2017 05:31 AM, Nisha Agarwal wrote:

Hi Matt,

As i understand John's spec https://review.openstack.org/#/c/507052/ <
https://review.openstack.org/#/c/507052/, it is actually a replacement
for capabilities(qualitative only) for ironic. It doesnt cover the
quantitative capabilities as 'traits' are meant only for qualitative
capabilities. Quantitative capabilities are covered by resource classes in
Nova. We have few (one or two) quantitative capabilities already supported
in ironic.

Hi Nisha,

This may be a case of mixed terminology. We do not refer to anything
quantitative as a "capability". Rather, we use the term "resource class"
(or sometimes just "resource") to represent quantitative things that may be
consumed by the instance.

Traits, on the other hand, are qualitative. They represent a binary on/off
capability that the compute host (or baremetal node in the case of Ironic)
exposes.

There's no limit on the number of traits that may be associated with a
particular Ironic baremetal node. However, for Ironic baremetal nodes, if
the node.resourceclass attribute is set, the Nova Ironic virt driver will
create a single inventory record for the node containing a quantity of 1
and a resource class equal to whatever is in the node.resource
class
attribute. This resource class is auto-created by Nova as a custom resource
class.

Just to follow up on this one...

I hope my traits spec will replace the need for the non-exact filters.

Consider two flavors Gold and GoldPlus. Lets say Goldplus gives you a
slightly newer CPU, or something.

Consider this setup:

  • both GOLD and GOLDPLUS ironic nodes have Resource Class: CUSTOMGOLD
  • but you can have some have trait: GOLDREGULAR and some with GOLDPLUS

Now you can have the flavors:

  • GOLD flavor requests resources:CUSTOM_GOLD=1
  • GOLDPLUS flavor also has resources:CUSTOMGOLD=1 but also
    trait:GOLD_PLUS:requires

Now eventually we could modify the GOLD flavor to say:

  • resources:CUSTOMGOLD=1 and trait:GOLDREGULAR:prefer

@Nisha I think that should largely allow you to construct the same behavior
you have today, or am I totally missing what you are wanting to do?

Thanks,
John


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by John_Garbutt (15,460 points)   3 4 5
0 votes

In Nisha's message, "capabilities" refers to "ComputeCapabilitiesFilter".
"capabilities" provides a lot of flexibility for scheduling. It supports
qualitative as well as quantitative attributes. It supports a variety of
operators such as ">=", "<", "=", etc. For instance, with "capabilities",
one can create a flavor for "GPU_Count >=2". Quantity matters for
workloads. A workload may require at least 2GPUs or at least certain
amount of SSD capacity to meet the performance requirements. Trait will
help but it only supports qualitative attributes. Therefore, we still need
"capabilities".

Standard Resource Class represents quantitative things but it 's not
available for Ironic. Ironic currently can only use one single
CUSTOMResourceclass with exact match. Prior to Pike, Ironic customers
can use non-exact match filter to support the use case of "at least this
amount of quantitative things on a bare metal node" but it's not supported
by Ironic Customresourceclass. Therefore it is a regression and will
cause issues for those who are depending on it.

On 19 October 2017 at 15:38, Jay Pipes jaypipes@gmail.com wrote:

On 10/16/2017 05:31 AM, Nisha Agarwal wrote:

Hi Matt,

As i understand John's spec https://review.openstack.org/#/c/507052/ <
https://review.openstack.org/#/c/507052/, it is actually a replacement
for capabilities(qualitative only) for ironic. It doesnt cover the
quantitative capabilities as 'traits' are meant only for qualitative
capabilities. Quantitative capabilities are covered by resource classes in
Nova. We have few (one or two) quantitative capabilities already supported
in ironic.

Hi Nisha,

This may be a case of mixed terminology. We do not refer to anything
quantitative as a "capability". Rather, we use the term "resource class"
(or sometimes just "resource") to represent quantitative things that may be
consumed by the instance.

Traits, on the other hand, are qualitative. They represent a binary on/off
capability that the compute host (or baremetal node in the case of Ironic)
exposes.

There's no limit on the number of traits that may be associated with a
particular Ironic baremetal node. However, for Ironic baremetal nodes, if
the node.resourceclass attribute is set, the Nova Ironic virt driver will
create a single inventory record for the node containing a quantity of 1
and a resource class equal to whatever is in the node.resource
class
attribute. This resource class is auto-created by Nova as a custom resource
class.

Just to follow up on this one...

I hope my traits spec will replace the need for the non-exact filters.

Consider two flavors Gold and GoldPlus. Lets say Goldplus gives you a
slightly newer CPU, or something.

Consider this setup:

  • both GOLD and GOLDPLUS ironic nodes have Resource Class: CUSTOMGOLD
  • but you can have some have trait: GOLDREGULAR and some with GOLDPLUS

Now you can have the flavors:

  • GOLD flavor requests resources:CUSTOM_GOLD=1
  • GOLDPLUS flavor also has resources:CUSTOMGOLD=1 but also
    trait:GOLD_PLUS:requires

Now eventually we could modify the GOLD flavor to say:

  • resources:CUSTOMGOLD=1 and trait:GOLDREGULAR:prefer

@Nisha I think that should largely allow you to construct the same behavior
you have today, or am I totally missing what you are wanting to do?

On Thu, Oct 19, 2017 at 8:37 AM, John Garbutt john@johngarbutt.com wrote:

On 19 October 2017 at 15:38, Jay Pipes jaypipes@gmail.com wrote:

On 10/16/2017 05:31 AM, Nisha Agarwal wrote:

Hi Matt,

As i understand John's spec https://review.openstack.org/#/c/507052/ <
https://review.openstack.org/#/c/507052/, it is actually a replacement
for capabilities(qualitative only) for ironic. It doesnt cover the
quantitative capabilities as 'traits' are meant only for qualitative
capabilities. Quantitative capabilities are covered by resource classes in
Nova. We have few (one or two) quantitative capabilities already supported
in ironic.

Hi Nisha,

This may be a case of mixed terminology. We do not refer to anything
quantitative as a "capability". Rather, we use the term "resource class"
(or sometimes just "resource") to represent quantitative things that may be
consumed by the instance.

Traits, on the other hand, are qualitative. They represent a binary
on/off capability that the compute host (or baremetal node in the case of
Ironic) exposes.

There's no limit on the number of traits that may be associated with a
particular Ironic baremetal node. However, for Ironic baremetal nodes, if
the node.resourceclass attribute is set, the Nova Ironic virt driver will
create a single inventory record for the node containing a quantity of 1
and a resource class equal to whatever is in the node.resource
class
attribute. This resource class is auto-created by Nova as a custom resource
class.

Just to follow up on this one...

I hope my traits spec will replace the need for the non-exact filters.

Consider two flavors Gold and GoldPlus. Lets say Goldplus gives you a
slightly newer CPU, or something.

Consider this setup:

  • both GOLD and GOLDPLUS ironic nodes have Resource Class: CUSTOMGOLD
  • but you can have some have trait: GOLDREGULAR and some with GOLDPLUS

Now you can have the flavors:

  • GOLD flavor requests resources:CUSTOM_GOLD=1
  • GOLDPLUS flavor also has resources:CUSTOMGOLD=1 but also
    trait:GOLD_PLUS:requires

Now eventually we could modify the GOLD flavor to say:

  • resources:CUSTOMGOLD=1 and trait:GOLDREGULAR:prefer

@Nisha I think that should largely allow you to construct the same
behavior you have today, or am I totally missing what you are wanting to do?

Thanks,
John


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 26, 2017 by Wan-yen_Hsu (480 points)   1
0 votes

On Oct 26, 2017, at 6:57 PM, Wan-yen Hsu wanyenhsu@gmail.com wrote:

In Nisha's message, "capabilities" refers to "ComputeCapabilitiesFilter". "capabilities" provides a lot of flexibility for scheduling. It supports qualitative as well as quantitative attributes. It supports a variety of operators such as ">=", "<", "=", etc. For instance, with "capabilities", one can create a flavor for "GPU_Count >=2". Quantity matters for workloads. A workload may require at least 2GPUs or at least certain amount of SSD capacity to meet the performance requirements. Trait will help but it only supports qualitative attributes. Therefore, we still need "capabilities".

In your example, you would create a resource class that specifies the number of GPUs. If there is a machine with no GPU, it would be a different resource class than a machine with a GPU. Likewise, a machine with 2 GPUs would be a different class. This gives you the ability to match the request to the need. Saying you need a machine with at least 2 GPUs means that you could end up with a machine with 100 GPUs - ok, I know that's not realistic, but it illustrates my point. Each hardware combination is a separate resource class. If your workload requires 2 GPUs and SSD, there are a finite number of hardware combinations available. You pick the flavor (i.e., resource class) that matches your need.

-- Ed Leafe


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Oct 27, 2017 by Ed_Leafe (11,720 points)   1 3 4
0 votes

Hi John,

@Nisha I think that should largely allow you to construct the same
behavior you have today, or am I totally missing what you are wanting to do?

Yes, i agree that qualitative capabilities are covered by your spec and
will give us the behaviour as its today with your spec implemented.

Regards
Nisha

On Fri, Oct 27, 2017 at 8:12 AM, Ed Leafe ed@leafe.com wrote:

On Oct 26, 2017, at 6:57 PM, Wan-yen Hsu wanyenhsu@gmail.com wrote:

In Nisha's message, "capabilities" refers to
"ComputeCapabilitiesFilter". "capabilities" provides a lot of flexibility
for scheduling. It supports qualitative as well as quantitative
attributes. It supports a variety of operators such as ">=", "<", "=",
etc. For instance, with "capabilities", one can create a flavor for
"GPU_Count >=2". Quantity matters for workloads. A workload may require
at least 2GPUs or at least certain amount of SSD capacity to meet the
performance requirements. Trait will help but it only supports
qualitative attributes. Therefore, we still need "capabilities".

In your example, you would create a resource class that specifies the
number of GPUs. If there is a machine with no GPU, it would be a different
resource class than a machine with a GPU. Likewise, a machine with 2 GPUs
would be a different class. This gives you the ability to match the request
to the need. Saying you need a machine with at least 2 GPUs means that you
could end up with a machine with 100 GPUs - ok, I know that's not
realistic, but it illustrates my point. Each hardware combination is a
separate resource class. If your workload requires 2 GPUs and SSD, there
are a finite number of hardware combinations available. You pick the flavor
(i.e., resource class) that matches your need.

-- Ed Leafe


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
The Secret Of Success is learning how to use pain and pleasure, instead
of having pain and pleasure use you. If You do that you are in control
of your life. If you don't life controls you.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 27, 2017 by Nisha_Agarwal (600 points)   1
...