settingsLogin | Registersettings

[openstack-dev] [ironic] using ironic as a replacement for existing datacenter baremetal provisioning

0 votes

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)
List of concerns with ironic:

1.)Nova <-> ironic interactions are generally seem terrible?
-How to accept raid config and partitioning(?) from end users? Seems to not a yet agreed upon method between nova/ironic.
-How to run multiple conductors/nova-computes? Right now as far as I can tell all of ironic front-end by a single nova-compute, which I will have to manage via a cluster technology between two or mode nodes. Because of this and the way host-agregates work I am unable to expose fault domains for ironic instances (all of ironic can only be under a single AZ (the az that is assigned to the nova-compute node)). Unless I create multiple nova-compute servers and manage multiple independent ironic setups. This makes on-boarding/query of hardware capacity painful.
- Nova appears to be forcing a we are "compute" as long as "compute" is VMs, means that we will have a baremetal flavor explosion (ie the mismatch between baremetal and VM).
- This is a feeling I got from the ironic-nova cross project meeting in Austin. General exmaple goes back to raid config above. I can configure a single piece of hardware many different ways, but to fit into nova's world view I need to have many different flavors exposed to end-user. In this way many flavors can map back to a single piece of hardware with just a lsightly different configuration applied. So how am I suppose to do a single server with 6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems like I would need to pre-mark out servers that were going to be a specific raid level. Which means that I need to start managing additional sub-pools of hardware to just deal with how the end users wants the raid configured, this is pretty much a non-starter for us. I have not really heard of whats being done on this specific front.

2.) Inspector:
- IPA service doesn't gather port/switching information
- Inspection service doesn't process port/switching information, which means that it wont add it to ironic. Which makes managing network swinging of the host a non-starter. As I would inspect the host – then modify the ironic record to add the details about what port/switch the server is connected to from a different source. At that point why wouldn't I just onboard everything through the API?
- Doesn't grab hardware disk configurations, If the server has multiple raids (r1 + r5) only reports boot raid disk capacity.
- Inspection is geared towards using a different network and dnsmasq infrastructure than what is in use for ironic/neutron. Which also means that in order to not conflict with dhcp requests for servers in ironic I need to use different networks. Which also means I now need to handle swinging server ports between different networks.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure issues. So I can not work without a fork for onboard of servers due to the fact that IPMI modules aren't built for the kernel, so inspection can never match the node against ironic. Seems like currently functionality here is MVP for gate to work and to deploy images. But if you need to do firmware, bios-config, any other hardware specific features you are pretty much going to need to roll your own IPA image and IPA modules to do standard provisioning tasks.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I have seen purposes to try and fix this?), this is painful to manage with large numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this might be fixed in newer versions of ironic?), again if end users aren't suppose to consume ironic api's directly - this is painful to handle.
- Its very easy to get a node to fall off the staemachine rails (reboot a server while an image is being deployed to it), the only way I have seen to be able to fix this is to update the DB directly.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova – so how are end users suppose to consume the shell-in-box console?
- I have BMC that need specific configuration (some require SOL on com2, others on com1) this makes it pretty much impossible without per box overrides against the conductor hardcoded templates.
- Additionally it would be nice to default to having a provisioning kernel/image that was set as a single config option with per server overrides – rather than on each server. If we ever change the IPA image – that means at scale we would need to update thousands of ironic nodes.

What is ironic doing to monitor the hardware for failures? I assume the answer here is nothing and that we will need to make sure the images that we deploy are correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Overall the above concerns have me wondering if I am going crazy, or is ironic really not ready to take over datacenters baremetal provisioning (unless you have a very limited view of functionality that is needed in those datacenters).

Thoughts would be great,


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 6, 2016 in openstack-dev by Kris_G._Lindgren (7,740 points)   1 7 11
retagged Jan 25, 2017 by admin

22 Responses

0 votes

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)

Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova <-> ironic interactions are generally seem terrible?

I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost

-How to accept raid config and partitioning(?) from end users? Seems to not a yet agreed upon method between nova/ironic.

AFAIK accepting it from the users just isn't solved. Administrators
do have custom ramdisks that they boot to pre-configure RAID during
enrollment.

-How to run multiple conductors/nova-computes? Right now as far as I can tell all of ironic front-end by a single nova-compute, which I will have to manage via a cluster technology between two or mode nodes. Because of this and the way host-agregates work I am unable to expose fault domains for ironic instances (all of ironic can only be under a single AZ (the az that is assigned to the nova-compute node)). Unless I create multiple nova-compute servers and manage multiple independent ironic setups. This makes on-boarding/query of hardware capacity painful.

The nova-compute does almost nothing. It really just talks to the
scheduler to tell it what's going on in Ironic. If it dies, deploys
won't stop. You can run many many conductors and spread load and fault
tolerance among them easily. I think for multiple AZs though, you're
right, there's no way to expose that. Perhaps it can be done with cells,
which I think Rackspace's OnMetal uses (but I'll let them refute or
confirm that).

Seems like the virt driver could be taught to be AZ-aware and some
metadata in the server record could allow AZs to go through to Ironic.

  • Nova appears to be forcing a we are "compute" as long as "compute" is VMs, means that we will have a baremetal flavor explosion (ie the mismatch between baremetal and VM).

    • This is a feeling I got from the ironic-nova cross project meeting in Austin. General exmaple goes back to raid config above. I can configure a single piece of hardware many different ways, but to fit into nova's world view I need to have many different flavors exposed to end-user. In this way many flavors can map back to a single piece of hardware with just a lsightly different configuration applied. So how am I suppose to do a single server with 6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems like I would need to pre-mark out servers that were going to be a specific raid level. Which means that I need to start managing additional sub-pools of hardware to just deal with how the end users wants the raid configured, this is pretty much a non-starter for us. I have not really heard of whats being done on this specific front.

You got that right. Perhaps people are comfortable with this limitation.
It is at least simple.

2.) Inspector:
- IPA service doesn't gather port/switching information
- Inspection service doesn't process port/switching information, which means that it wont add it to ironic. Which makes managing network swinging of the host a non-starter. As I would inspect the host – then modify the ironic record to add the details about what port/switch the server is connected to from a different source. At that point why wouldn't I just onboard everything through the API?
- Doesn't grab hardware disk configurations, If the server has multiple raids (r1 + r5) only reports boot raid disk capacity.
- Inspection is geared towards using a different network and dnsmasq infrastructure than what is in use for ironic/neutron. Which also means that in order to not conflict with dhcp requests for servers in ironic I need to use different networks. Which also means I now need to handle swinging server ports between different networks.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure issues. So I can not work without a fork for onboard of servers due to the fact that IPMI modules aren't built for the kernel, so inspection can never match the node against ironic. Seems like currently functionality here is MVP for gate to work and to deploy images. But if you need to do firmware, bios-config, any other hardware specific features you are pretty much going to need to roll your own IPA image and IPA modules to do standard provisioning tasks.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I have seen purposes to try and fix this?), this is painful to manage with large numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this might be fixed in newer versions of ironic?), again if end users aren't suppose to consume ironic api's directly - this is painful to handle.
- Its very easy to get a node to fall off the staemachine rails (reboot a server while an image is being deployed to it), the only way I have seen to be able to fix this is to update the DB directly.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova – so how are end users suppose to consume the shell-in-box console?
- I have BMC that need specific configuration (some require SOL on com2, others on com1) this makes it pretty much impossible without per box overrides against the conductor hardcoded templates.
- Additionally it would be nice to default to having a provisioning kernel/image that was set as a single config option with per server overrides – rather than on each server. If we ever change the IPA image – that means at scale we would need to update thousands of ironic nodes.

What is ironic doing to monitor the hardware for failures? I assume the answer here is nothing and that we will need to make sure the images that we deploy are correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Overall the above concerns have me wondering if I am going crazy, or is ironic really not ready to take over datacenters baremetal provisioning (unless you have a very limited view of functionality that is needed in those datacenters).

You're not crazy at all. I remember having some of these discussions in
the early days when it was just 'nova baremetal' and people wanted AZs
and better serial console access. For whatever reason, it just hasn't
quite happened yet.

It's not a small job, but I don't think anything you say above is
particularly troubling. I'd certainly like better console support and AZs.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 6, 2016 by Clint_Byrum (40,940 points)   4 6 10
0 votes

On 06/06/2016 01:44 PM, Kris G. Lindgren wrote:
Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following
in an attempt to document some of my concerns, and I'm wondering if you folks
could help myself identity ongoing work to solve these (or alternatives?)
List of concerns with ironic:

Hi Kris,

There is a lot of ongoing work in and around the Ironic project. Thanks for
diving in and for sharing your concerns; you're not alone.

I'll respond to each group of concerns, as some of these appear quite similar to
each other and align with stuff we're already doing. Hopefully I can provide
some helpful background to where the project is at today.

1.)Nova <-> ironic interactions are generally seem terrible?

These two projects are coming at the task of managing "compute" with
significantly different situations and we've been working, for the last ~2
years, to build a framework that can provide both virtual and physical resources
through one API. It's not a simple task, and we have a lot more to do.

-How to accept raid config and partitioning(?) from end users? Seems to not a
yet agreed upon method between nova/ironic.

Nova expresses partitioning in a very limited way on the flavor. You get root,
swap, and ephemeral partitions -- and that's it. Ironic honors those today, but
they're pinned on the flavor definition, eg. by the cloud admin (or whoever can
define the flavor.

If your users need more complex partitioning, they could create additional
partitions after the instance is created. This limitation within Ironic exists,
in part, because the projects' goal is to provide hardware through the OpenStack
Compute API -- which doesn't express arbitrary partitionability. (If you're
interested, there is a lengthier and more political discussion about whether the
cloud should support "pets" and whether arbitrary partitioning is needed for
"cattle".)

RAID configuration isn't something that Nova allows their users to choose today
- it doesn't fit in the Nova model of "compute", and there is, to my knowledge,
nothing in the Nova API to allow its input. We've discussed this a little bit,
but so far settled on leaving it up to the cloud admin to set this in Ironic.

There has been discussion with the Cinder community over ways to express volume
spanning and mirroring, but apply it to a machines' local disks, but these
discussions didn't result in any traction.

There's also been discussion of ways we could do ad-hoc changes in RAID level,
based on flavor metadata, during the provisioning process (rather than ahead of
time) but no code has been done for this yet, AFAIK.

So, where does that leave us? With the "explosion of flavors" that you
described. It may not be ideal, but that is the common ground we've reached.

-How to run multiple conductors/nova-computes? Right now as far as I can
tell all of ironic front-end by a single nova-compute, which I will have to
manage via a cluster technology between two or mode nodes. Because of this and
the way host-agregates work I am unable to expose fault domains for ironic
instances (all of ironic can only be under a single AZ (the az that is assigned
to the nova-compute node)). Unless I create multiple nova-compute servers and
manage multiple independent ironic setups. This makes on-boarding/query of
hardware capacity painful.

Yep. It's not ideal, and the community is very well aware of, and actively
working on, this limitation. It also may not be as bad as you may think. The
nova-compute process doesn't do very much, and tests show it handling some
thousands of ironic nodes fairly well in parallel. Standard active-passive
management of that process should suffice.

A lot of design work has been done to come up with a joint solution by folks on
both the Ironic and Nova teams.
http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html

As a side note, it's possible (though not tested, recommended, or well
documented) to run more than one nova-compute. See
https://github.com/openstack/ironic/blob/master/ironic/nova/compute/manager.py

  • Nova appears to be forcing a we are "compute" as long as "compute" is VMs,
    means that we will have a baremetal flavor explosion (ie the mismatch between
    baremetal and VM).

    • This is a feeling I got from the ironic-nova cross project meeting in
      Austin. General exmaple goes back to raid config above. I can configure a
      single piece of hardware many different ways, but to fit into nova's world view
      I need to have many different flavors exposed to end-user. In this way many
      flavors can map back to a single piece of hardware with just a lsightly
      different configuration applied. So how am I suppose to do a single server with
      6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems
      like I would need to pre-mark out servers that were going to be a specific raid
      level. Which means that I need to start managing additional sub-pools of
      hardware to just deal with how the end users wants the raid configured, this is
      pretty much a non-starter for us. I have not really heard of whats being done
      on this specific front.

You're correct. Again, Nova has no concept of RAID in their API, so yea, today
you're left with a 'flavor explosion', as you put it.

There's been discussion of methods we could use to apply the RAID level during
provisioning, but generally those discussions have landed on the side of "it's
the operators responsibility to maintain pools of resources available that match
their customers' demand".

2.) Inspector:
- IPA service doesn't gather port/switching information

Folks are working on this, but it's been blocked for a while on the
ironic-neutron integration:
https://review.openstack.org/#/c/241242/

  • Inspection service doesn't process port/switching information, which means
    that it wont add it to ironic. Which makes managing network swinging of the
    host a non-starter. As I would inspect the host – then modify the ironic record
    to add the details about what port/switch the server is connected to from a
    different source. At that point why wouldn't I just onboard everything through
    the API?

This is desired, but not done yet, AFAIK.

  • Doesn't grab hardware disk configurations, If the server has multiple raids
    (r1 + r5) only reports boot raid disk capacity.

This falls out from a limitation in Nova (discussed above) though I would
encourage inspector to collect all the data (even if ironic/nova can't use it,
today).

  • Inspection is geared towards using a different network and dnsmasq
    infrastructure than what is in use for ironic/neutron. Which also means that in
    order to not conflict with dhcp requests for servers in ironic I need to use
    different networks. Which also means I now need to handle swinging server ports
    between different networks.

Inspector is designed to respond only to requests for nodes in the inspection
phase, so that it doesn't conflict with provisioning of nodes by Ironic. I've
been using the same network for inspection and provisioning without issue -- so
I'm not sure what problem you're encountering here.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure
issues. So I can not work without a fork for onboard of servers due to the fact
that IPMI modules aren't built for the kernel, so inspection can never match the
node against ironic. Seems like currently functionality here is MVP for gate to
work and to deploy images. But if you need to do firmware, bios-config, any
other hardware specific features you are pretty much going to need to roll your
own IPA image and IPA modules to do standard provisioning tasks.

That's correct. We assume that operators and downstream distributors will build
and customize the IPA image as needed for their environment. Ironic only
provides the base image and the tools to modify it; if we were to attempt to
build an image that could handle every piece of hardware out there, it would be
huge, unwieldy, and contain a lot of proprietary tools that we simply don't have
access / license to use.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I
have seen purposes to try and fix this?), this is painful to manage with large
numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this
might be fixed in newer versions of ironic?), again if end users aren't suppose
to consume ironic api's directly - this is painful to handle.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova –
so how are end users suppose to consume the shell-in-box console?

You are, unfortunately, correct. Ironic once supported SOL console connectivity
through Nova, but it has not been working for a while now. We discussed this at
length in the Austin summit and plan to fix it soon:
https://review.openstack.org/#/c/319505/

  • Its very easy to get a node to fall off the staemachine rails (reboot a
    server while an image is being deployed to it), the only way I have seen to be
    able to fix this is to update the DB directly.

Yea, that's a well known pain point, and there is ongoing work to improve the
recovery process for nodes that get "stuck" in various ways, with the premise
that the operator should never have to munge the DB directly. One approach we've
discussed is adding a management CLI tool to make this cleaner.

  • I have BMC that need specific configuration (some require SOL on com2,
    others on com1) this makes it pretty much impossible without per box overrides
    against the conductor hardcoded templates.

Ironic allows certain aspects of the Node's management to be overridden
individually, but it sounds like you need some knobs that we haven't
implemented. Could you file a bug for this? I think we'd be keen to add it.

  • Additionally it would be nice to default to having a provisioning
    kernel/image that was set as a single config option with per server overrides –
    rather than on each server. If we ever change the IPA image – that means at
    scale we would need to update thousands of ironic nodes.

This request has surfaced in the past, however, it wouldn't make sense in a
heterogeneous environment (eg, mix of ia64 and x86_64 hardware in one region)
and so past discussions have landed on the side of not implementing it (either
as a system-level default image or as a driver-level default image).

If there were a consensus that it helped enough deployments, without increasing
the complexity of complex multi-arch deployments, I think folks would be willing
to accept a feature like this.

What is ironic doing to monitor the hardware for failures? I assume the answer
here is nothing and that we will need to make sure the images that we deploy are
correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Today, nothing, but this is something we want to do, and there is an agreed-upon
design, here:
http://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/notifications.html

This is only the initial design for the notification system. The goal of this
would be to enable drivers to capture hardware alerts (or perform more proactive
gathering of hardware status) and propagate those alerts up to the cloud operator.

In summary, you're not alone, nor are your ideas/thoughts/requests unreasonable.
We're all facing similar concerns -- and you're welcome to come hang out in

openstack-ironic and participate in shaping Ironic so that it meets your needs,

too :)

Regards,
Devananda


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jun 7, 2016 by Devananda_van_der_Ve (10,380 points)   2 4 6
0 votes

Thanks for getting to this before me, Deva. Saved me some typing. :)

A little more color inline.

On Mon, Jun 06, 2016 at 05:01:04PM -0700, Devananda van der Veen wrote:

On 06/06/2016 01:44 PM, Kris G. Lindgren wrote:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following
in an attempt to document some of my concerns, and I'm wondering if you folks
could help myself identity ongoing work to solve these (or alternatives?)
List of concerns with ironic:

Hi Kris,

There is a lot of ongoing work in and around the Ironic project. Thanks for
diving in and for sharing your concerns; you're not alone.

I'll respond to each group of concerns, as some of these appear quite similar to
each other and align with stuff we're already doing. Hopefully I can provide
some helpful background to where the project is at today.

1.)Nova <-> ironic interactions are generally seem terrible?

These two projects are coming at the task of managing "compute" with
significantly different situations and we've been working, for the last ~2
years, to build a framework that can provide both virtual and physical resources
through one API. It's not a simple task, and we have a lot more to do.

-How to accept raid config and partitioning(?) from end users? Seems to not a
yet agreed upon method between nova/ironic.

Nova expresses partitioning in a very limited way on the flavor. You get root,
swap, and ephemeral partitions -- and that's it. Ironic honors those today, but
they're pinned on the flavor definition, eg. by the cloud admin (or whoever can
define the flavor.

If your users need more complex partitioning, they could create additional
partitions after the instance is created. This limitation within Ironic exists,
in part, because the projects' goal is to provide hardware through the OpenStack
Compute API -- which doesn't express arbitrary partitionability. (If you're
interested, there is a lengthier and more political discussion about whether the
cloud should support "pets" and whether arbitrary partitioning is needed for
"cattle".)

RAID configuration isn't something that Nova allows their users to choose today
- it doesn't fit in the Nova model of "compute", and there is, to my knowledge,
nothing in the Nova API to allow its input. We've discussed this a little bit,
but so far settled on leaving it up to the cloud admin to set this in Ironic.

There has been discussion with the Cinder community over ways to express volume
spanning and mirroring, but apply it to a machines' local disks, but these
discussions didn't result in any traction.

There's also been discussion of ways we could do ad-hoc changes in RAID level,
based on flavor metadata, during the provisioning process (rather than ahead of
time) but no code has been done for this yet, AFAIK.

So, where does that leave us? With the "explosion of flavors" that you
described. It may not be ideal, but that is the common ground we've reached.

-How to run multiple conductors/nova-computes? Right now as far as I can
tell all of ironic front-end by a single nova-compute, which I will have to
manage via a cluster technology between two or mode nodes. Because of this and
the way host-agregates work I am unable to expose fault domains for ironic
instances (all of ironic can only be under a single AZ (the az that is assigned
to the nova-compute node)). Unless I create multiple nova-compute servers and
manage multiple independent ironic setups. This makes on-boarding/query of
hardware capacity painful.

Yep. It's not ideal, and the community is very well aware of, and actively
working on, this limitation. It also may not be as bad as you may think. The
nova-compute process doesn't do very much, and tests show it handling some
thousands of ironic nodes fairly well in parallel. Standard active-passive
management of that process should suffice.

A lot of design work has been done to come up with a joint solution by folks on
both the Ironic and Nova teams.
http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html

It's important to point out here that we're re-working how this works,
but it's still one of our highest priorities:
https://review.openstack.org/#/c/320016/

As a side note, it's possible (though not tested, recommended, or well
documented) to run more than one nova-compute. See
https://github.com/openstack/ironic/blob/master/ironic/nova/compute/manager.py

  • Nova appears to be forcing a we are "compute" as long as "compute" is VMs,
    means that we will have a baremetal flavor explosion (ie the mismatch between
    baremetal and VM).

    • This is a feeling I got from the ironic-nova cross project meeting in
      Austin. General exmaple goes back to raid config above. I can configure a
      single piece of hardware many different ways, but to fit into nova's world view
      I need to have many different flavors exposed to end-user. In this way many
      flavors can map back to a single piece of hardware with just a lsightly
      different configuration applied. So how am I suppose to do a single server with
      6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems
      like I would need to pre-mark out servers that were going to be a specific raid
      level. Which means that I need to start managing additional sub-pools of
      hardware to just deal with how the end users wants the raid configured, this is
      pretty much a non-starter for us. I have not really heard of whats being done
      on this specific front.

You're correct. Again, Nova has no concept of RAID in their API, so yea, today
you're left with a 'flavor explosion', as you put it.

There's been discussion of methods we could use to apply the RAID level during
provisioning, but generally those discussions have landed on the side of "it's
the operators responsibility to maintain pools of resources available that match
their customers' demand".

2.) Inspector:
- IPA service doesn't gather port/switching information

Folks are working on this, but it's been blocked for a while on the
ironic-neutron integration:
https://review.openstack.org/#/c/241242/

  • Inspection service doesn't process port/switching information, which means
    that it wont add it to ironic. Which makes managing network swinging of the
    host a non-starter. As I would inspect the host – then modify the ironic record
    to add the details about what port/switch the server is connected to from a
    different source. At that point why wouldn't I just onboard everything through
    the API?

This is desired, but not done yet, AFAIK.

  • Doesn't grab hardware disk configurations, If the server has multiple raids
    (r1 + r5) only reports boot raid disk capacity.

This falls out from a limitation in Nova (discussed above) though I would
encourage inspector to collect all the data (even if ironic/nova can't use it,
today).

  • Inspection is geared towards using a different network and dnsmasq
    infrastructure than what is in use for ironic/neutron. Which also means that in
    order to not conflict with dhcp requests for servers in ironic I need to use
    different networks. Which also means I now need to handle swinging server ports
    between different networks.

Inspector is designed to respond only to requests for nodes in the inspection
phase, so that it doesn't conflict with provisioning of nodes by Ironic. I've
been using the same network for inspection and provisioning without issue -- so
I'm not sure what problem you're encountering here.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure
issues. So I can not work without a fork for onboard of servers due to the fact
that IPMI modules aren't built for the kernel, so inspection can never match the
node against ironic. Seems like currently functionality here is MVP for gate to
work and to deploy images. But if you need to do firmware, bios-config, any
other hardware specific features you are pretty much going to need to roll your
own IPA image and IPA modules to do standard provisioning tasks.

That's correct. We assume that operators and downstream distributors will build
and customize the IPA image as needed for their environment. Ironic only
provides the base image and the tools to modify it; if we were to attempt to
build an image that could handle every piece of hardware out there, it would be
huge, unwieldy, and contain a lot of proprietary tools that we simply don't have
access / license to use.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I
have seen purposes to try and fix this?), this is painful to manage with large
numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this
might be fixed in newer versions of ironic?), again if end users aren't suppose
to consume ironic api's directly - this is painful to handle.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova –
so how are end users suppose to consume the shell-in-box console?

You are, unfortunately, correct. Ironic once supported SOL console connectivity
through Nova, but it has not been working for a while now. We discussed this at
length in the Austin summit and plan to fix it soon:
https://review.openstack.org/#/c/319505/

  • Its very easy to get a node to fall off the staemachine rails (reboot a
    server while an image is being deployed to it), the only way I have seen to be
    able to fix this is to update the DB directly.

Yea, that's a well known pain point, and there is ongoing work to improve the
recovery process for nodes that get "stuck" in various ways, with the premise
that the operator should never have to munge the DB directly. One approach we've
discussed is adding a management CLI tool to make this cleaner.

  • I have BMC that need specific configuration (some require SOL on com2,
    others on com1) this makes it pretty much impossible without per box overrides
    against the conductor hardcoded templates.

Ironic allows certain aspects of the Node's management to be overridden
individually, but it sounds like you need some knobs that we haven't
implemented. Could you file a bug for this? I think we'd be keen to add it.

Yeah, we've talked about this before but nobody has really pushed on it.
Essentially an optional pxe_append_params per node. Shouldn't be too
hard to implement.

  • Additionally it would be nice to default to having a provisioning
    kernel/image that was set as a single config option with per server overrides –
    rather than on each server. If we ever change the IPA image – that means at
    scale we would need to update thousands of ironic nodes.

This request has surfaced in the past, however, it wouldn't make sense in a
heterogeneous environment (eg, mix of ia64 and x86_64 hardware in one region)
and so past discussions have landed on the side of not implementing it (either
as a system-level default image or as a driver-level default image).

If there were a consensus that it helped enough deployments, without increasing
the complexity of complex multi-arch deployments, I think folks would be willing
to accept a feature like this.

This wouldn't be too complex, right? A config for the default deploy
image, overridden per-node in driver_info (like we do today). Multi-arch
deployments, in the worst case, would behave just like today, so I don't
see a problem here.

I suspect most single-arch deployments use the same ramdisk everywhere,
so this would be a huge help. The only downside I see right now is
needing to restart the conductor to roll a new ramdisk out, but that's a
pretty uneventful thing if you're running multiple conductors.

What is ironic doing to monitor the hardware for failures? I assume the answer
here is nothing and that we will need to make sure the images that we deploy are
correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Today, nothing, but this is something we want to do, and there is an agreed-upon
design, here:
http://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/notifications.html

This is only the initial design for the notification system. The goal of this
would be to enable drivers to capture hardware alerts (or perform more proactive
gathering of hardware status) and propagate those alerts up to the cloud operator.

In summary, you're not alone, nor are your ideas/thoughts/requests unreasonable.
We're all facing similar concerns -- and you're welcome to come hang out in

openstack-ironic and participate in shaping Ironic so that it meets your needs,

too :)

++

// jim

Regards,
Devananda


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by Jim_Rollenhagen (12,800 points)   2 3 3
0 votes

On 06/07/2016 02:01 AM, Devananda van der Veen wrote:

On 06/06/2016 01:44 PM, Kris G. Lindgren wrote:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following
in an attempt to document some of my concerns, and I'm wondering if you folks
could help myself identity ongoing work to solve these (or alternatives?)
List of concerns with ironic:

Hi Kris,

There is a lot of ongoing work in and around the Ironic project. Thanks for
diving in and for sharing your concerns; you're not alone.

I'll respond to each group of concerns, as some of these appear quite similar to
each other and align with stuff we're already doing. Hopefully I can provide
some helpful background to where the project is at today.

1.)Nova <-> ironic interactions are generally seem terrible?

These two projects are coming at the task of managing "compute" with
significantly different situations and we've been working, for the last ~2
years, to build a framework that can provide both virtual and physical resources
through one API. It's not a simple task, and we have a lot more to do.

-How to accept raid config and partitioning(?) from end users? Seems to not a
yet agreed upon method between nova/ironic.

Nova expresses partitioning in a very limited way on the flavor. You get root,
swap, and ephemeral partitions -- and that's it. Ironic honors those today, but
they're pinned on the flavor definition, eg. by the cloud admin (or whoever can
define the flavor.

If your users need more complex partitioning, they could create additional
partitions after the instance is created. This limitation within Ironic exists,
in part, because the projects' goal is to provide hardware through the OpenStack
Compute API -- which doesn't express arbitrary partitionability. (If you're
interested, there is a lengthier and more political discussion about whether the
cloud should support "pets" and whether arbitrary partitioning is needed for
"cattle".)

RAID configuration isn't something that Nova allows their users to choose today
- it doesn't fit in the Nova model of "compute", and there is, to my knowledge,
nothing in the Nova API to allow its input. We've discussed this a little bit,
but so far settled on leaving it up to the cloud admin to set this in Ironic.

There has been discussion with the Cinder community over ways to express volume
spanning and mirroring, but apply it to a machines' local disks, but these
discussions didn't result in any traction.

There's also been discussion of ways we could do ad-hoc changes in RAID level,
based on flavor metadata, during the provisioning process (rather than ahead of
time) but no code has been done for this yet, AFAIK.

I'm still pretty interested in it, because I agree with anything said
above about building RAID ahead-of-time not being convenient. I don't
quite understand how such a feature would look like, we might add it as
a topic for midcycle.

So, where does that leave us? With the "explosion of flavors" that you
described. It may not be ideal, but that is the common ground we've reached.

-How to run multiple conductors/nova-computes? Right now as far as I can
tell all of ironic front-end by a single nova-compute, which I will have to
manage via a cluster technology between two or mode nodes. Because of this and
the way host-agregates work I am unable to expose fault domains for ironic
instances (all of ironic can only be under a single AZ (the az that is assigned
to the nova-compute node)). Unless I create multiple nova-compute servers and
manage multiple independent ironic setups. This makes on-boarding/query of
hardware capacity painful.

Yep. It's not ideal, and the community is very well aware of, and actively
working on, this limitation. It also may not be as bad as you may think. The
nova-compute process doesn't do very much, and tests show it handling some
thousands of ironic nodes fairly well in parallel. Standard active-passive
management of that process should suffice.

A lot of design work has been done to come up with a joint solution by folks on
both the Ironic and Nova teams.
http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html

As a side note, it's possible (though not tested, recommended, or well
documented) to run more than one nova-compute. See
https://github.com/openstack/ironic/blob/master/ironic/nova/compute/manager.py

  • Nova appears to be forcing a we are "compute" as long as "compute" is VMs,
    means that we will have a baremetal flavor explosion (ie the mismatch between
    baremetal and VM).

    • This is a feeling I got from the ironic-nova cross project meeting in
      Austin. General exmaple goes back to raid config above. I can configure a
      single piece of hardware many different ways, but to fit into nova's world view
      I need to have many different flavors exposed to end-user. In this way many
      flavors can map back to a single piece of hardware with just a lsightly
      different configuration applied. So how am I suppose to do a single server with
      6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems
      like I would need to pre-mark out servers that were going to be a specific raid
      level. Which means that I need to start managing additional sub-pools of
      hardware to just deal with how the end users wants the raid configured, this is
      pretty much a non-starter for us. I have not really heard of whats being done
      on this specific front.

You're correct. Again, Nova has no concept of RAID in their API, so yea, today
you're left with a 'flavor explosion', as you put it.

There's been discussion of methods we could use to apply the RAID level during
provisioning, but generally those discussions have landed on the side of "it's
the operators responsibility to maintain pools of resources available that match
their customers' demand".

2.) Inspector:
- IPA service doesn't gather port/switching information

Folks are working on this, but it's been blocked for a while on the
ironic-neutron integration:
https://review.openstack.org/#/c/241242/

  • Inspection service doesn't process port/switching information, which means
    that it wont add it to ironic. Which makes managing network swinging of the
    host a non-starter. As I would inspect the host – then modify the ironic record
    to add the details about what port/switch the server is connected to from a
    different source. At that point why wouldn't I just onboard everything through
    the API?

This is desired, but not done yet, AFAIK.

  • Doesn't grab hardware disk configurations, If the server has multiple raids
    (r1 + r5) only reports boot raid disk capacity.

This falls out from a limitation in Nova (discussed above) though I would
encourage inspector to collect all the data (even if ironic/nova can't use it,
today).

  • Inspection is geared towards using a different network and dnsmasq
    infrastructure than what is in use for ironic/neutron. Which also means that in
    order to not conflict with dhcp requests for servers in ironic I need to use
    different networks. Which also means I now need to handle swinging server ports
    between different networks.

Inspector is designed to respond only to requests for nodes in the inspection
phase, so that it doesn't conflict with provisioning of nodes by Ironic. I've
been using the same network for inspection and provisioning without issue -- so
I'm not sure what problem you're encountering here.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure
issues. So I can not work without a fork for onboard of servers due to the fact
that IPMI modules aren't built for the kernel, so inspection can never match the
node against ironic. Seems like currently functionality here is MVP for gate to
work and to deploy images. But if you need to do firmware, bios-config, any
other hardware specific features you are pretty much going to need to roll your
own IPA image and IPA modules to do standard provisioning tasks.

That's correct. We assume that operators and downstream distributors will build
and customize the IPA image as needed for their environment. Ironic only
provides the base image and the tools to modify it; if we were to attempt to
build an image that could handle every piece of hardware out there, it would be
huge, unwieldy, and contain a lot of proprietary tools that we simply don't have
access / license to use.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I
have seen purposes to try and fix this?), this is painful to manage with large
numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this
might be fixed in newer versions of ironic?), again if end users aren't suppose
to consume ironic api's directly - this is painful to handle.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova –
so how are end users suppose to consume the shell-in-box console?

You are, unfortunately, correct. Ironic once supported SOL console connectivity
through Nova, but it has not been working for a while now. We discussed this at
length in the Austin summit and plan to fix it soon:
https://review.openstack.org/#/c/319505/

  • Its very easy to get a node to fall off the staemachine rails (reboot a
    server while an image is being deployed to it), the only way I have seen to be
    able to fix this is to update the DB directly.

Yea, that's a well known pain point, and there is ongoing work to improve the
recovery process for nodes that get "stuck" in various ways, with the premise
that the operator should never have to munge the DB directly. One approach we've
discussed is adding a management CLI tool to make this cleaner.

  • I have BMC that need specific configuration (some require SOL on com2,
    others on com1) this makes it pretty much impossible without per box overrides
    against the conductor hardcoded templates.

Ironic allows certain aspects of the Node's management to be overridden
individually, but it sounds like you need some knobs that we haven't
implemented. Could you file a bug for this? I think we'd be keen to add it.

  • Additionally it would be nice to default to having a provisioning
    kernel/image that was set as a single config option with per server overrides –
    rather than on each server. If we ever change the IPA image – that means at
    scale we would need to update thousands of ironic nodes.

This request has surfaced in the past, however, it wouldn't make sense in a
heterogeneous environment (eg, mix of ia64 and x86_64 hardware in one region)
and so past discussions have landed on the side of not implementing it (either
as a system-level default image or as a driver-level default image).

If there were a consensus that it helped enough deployments, without increasing
the complexity of complex multi-arch deployments, I think folks would be willing
to accept a feature like this.

What is ironic doing to monitor the hardware for failures? I assume the answer
here is nothing and that we will need to make sure the images that we deploy are
correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Today, nothing, but this is something we want to do, and there is an agreed-upon
design, here:
http://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/notifications.html

This is only the initial design for the notification system. The goal of this
would be to enable drivers to capture hardware alerts (or perform more proactive
gathering of hardware status) and propagate those alerts up to the cloud operator.

In summary, you're not alone, nor are your ideas/thoughts/requests unreasonable.
We're all facing similar concerns -- and you're welcome to come hang out in

openstack-ironic and participate in shaping Ironic so that it meets your needs,

too :)

Regards,
Devananda


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by Dmitry_Tantsur (18,080 points)   2 3 8
0 votes

Clint Byrum wrote:
Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)

Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?

I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost

Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?

That seems almost easier than building a bunch of ansible scripts that
appear (at a glance) to do similar things; and u get the benefit of
using a actual programming language vs a
half-programming-ansible-yaml-language...

A realization I'm having is that I'm really not a fan of using ansible
as a half-programming-ansible-yaml-language, which it seems like people
start to try to do after a while (because at some point you need
something like if statements, then things like [1] get created), no
offense to the authors, but I guess this is my personal preference (it's
also one of the reasons taskflow directly is a lib. in python, because
, people don't need to learn a new language).

[1]
https://github.com/openstack/bifrost/blob/master/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml

-How to accept raid config and partitioning(?) from end users? Seems to not a yet agreed upon method between nova/ironic.

AFAIK accepting it from the users just isn't solved. Administrators
do have custom ramdisks that they boot to pre-configure RAID during
enrollment.

-How to run multiple conductors/nova-computes?   Right now as far as I can tell all of ironic front-end by a single nova-compute, which I will have to manage via a cluster technology between two or mode nodes.  Because of this and the way host-agregates work I am unable to expose fault domains for ironic instances (all of ironic can only be under a single AZ (the az that is assigned to the nova-compute node)). Unless I create multiple nova-compute servers and manage multiple independent ironic setups.  This makes on-boarding/query of hardware capacity painful.

The nova-compute does almost nothing. It really just talks to the
scheduler to tell it what's going on in Ironic. If it dies, deploys
won't stop. You can run many many conductors and spread load and fault
tolerance among them easily. I think for multiple AZs though, you're
right, there's no way to expose that. Perhaps it can be done with cells,
which I think Rackspace's OnMetal uses (but I'll let them refute or
confirm that).

Seems like the virt driver could be taught to be AZ-aware and some
metadata in the server record could allow AZs to go through to Ironic.

  • Nova appears to be forcing a we are "compute" as long as "compute" is VMs, means that we will have a baremetal flavor explosion (ie the mismatch between baremetal and VM).

    • This is a feeling I got from the ironic-nova cross project meeting in Austin. General exmaple goes back to raid config above. I can configure a single piece of hardware many different ways, but to fit into nova's world view I need to have many different flavors exposed to end-user. In this way many flavors can map back to a single piece of hardware with just a lsightly different configuration applied. So how am I suppose to do a single server with 6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD. Seems like I would need to pre-mark out servers that were going to be a specific raid level. Which means that I need to start managing additional sub-pools of hardware to just deal with how the end users wants the raid configured, this is pretty much a non-starter for us. I have not really heard of whats being done on this specific front.

You got that right. Perhaps people are comfortable with this limitation.
It is at least simple.

2.) Inspector:
- IPA service doesn't gather port/switching information
- Inspection service doesn't process port/switching information, which means that it wont add it to ironic. Which makes managing network swinging of the host a non-starter. As I would inspect the host – then modify the ironic record to add the details about what port/switch the server is connected to from a different source. At that point why wouldn't I just onboard everything through the API?
- Doesn't grab hardware disk configurations, If the server has multiple raids (r1 + r5) only reports boot raid disk capacity.
- Inspection is geared towards using a different network and dnsmasq infrastructure than what is in use for ironic/neutron. Which also means that in order to not conflict with dhcp requests for servers in ironic I need to use different networks. Which also means I now need to handle swinging server ports between different networks.

3.) IPA image:
- Default build stuff is pinned to extremly old versions due to gate failure issues. So I can not work without a fork for onboard of servers due to the fact that IPMI modules aren't built for the kernel, so inspection can never match the node against ironic. Seems like currently functionality here is MVP for gate to work and to deploy images. But if you need to do firmware, bios-config, any other hardware specific features you are pretty much going to need to roll your own IPA image and IPA modules to do standard provisioning tasks.

4.) Conductor:
- Serial-over-lan consoles require a unique port on the conductor server (I have seen purposes to try and fix this?), this is painful to manage with large numbers of servers.
- SOL consoles aren't restarted when conductor is restarted (I think this might be fixed in newer versions of ironic?), again if end users aren't suppose to consume ironic api's directly - this is painful to handle.
- Its very easy to get a node to fall off the staemachine rails (reboot a server while an image is being deployed to it), the only way I have seen to be able to fix this is to update the DB directly.
- As far as I can tell shell-in-a- box, SOL consoles aren't support via nova – so how are end users suppose to consume the shell-in-box console?
- I have BMC that need specific configuration (some require SOL on com2, others on com1) this makes it pretty much impossible without per box overrides against the conductor hardcoded templates.
- Additionally it would be nice to default to having a provisioning kernel/image that was set as a single config option with per server overrides – rather than on each server. If we ever change the IPA image – that means at scale we would need to update thousands of ironic nodes.

What is ironic doing to monitor the hardware for failures? I assume the answer here is nothing and that we will need to make sure the images that we deploy are correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.

Overall the above concerns have me wondering if I am going crazy, or is ironic really not ready to take over datacenters baremetal provisioning (unless you have a very limited view of functionality that is needed in those datacenters).

You're not crazy at all. I remember having some of these discussions in
the early days when it was just 'nova baremetal' and people wanted AZs
and better serial console access. For whatever reason, it just hasn't
quite happened yet.

It's not a small job, but I don't think anything you say above is
particularly troubling. I'd certainly like better console support and AZs.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by harlowja_at_fastmail (16,200 points)   2 6 8
0 votes

Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:

Clint Byrum wrote:

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)

Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?

I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost

Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?

That's an interesting idea. I think a reason Bifrost doesn't just import
nova virt drivers is that they're likely not a supported public API
(despite not having 's at the front). Also, a lot of the reason Bifrost
exists is to enable users to get the benefits of all the baremetal
abstraction work done in Ironic without having to fully embrace all of
OpenStack's core. So while you could get a little bit of the stuff from
nova (like config drive building), you'd still need to handle network
address assignment, image management, etc. etc., and pretty soon you
start having to run a tiny glance and a tiny neutron. The Bifrost way
is the opposite: I just want a tiny Ironic, and _nothing
else.

That seems almost easier than building a bunch of ansible scripts that
appear (at a glance) to do similar things; and u get the benefit of
using a actual programming language vs a
half-programming-ansible-yaml-language...

A realization I'm having is that I'm really not a fan of using ansible
as a half-programming-ansible-yaml-language, which it seems like people
start to try to do after a while (because at some point you need
something like if statements, then things like [1] get created), no
offense to the authors, but I guess this is my personal preference (it's
also one of the reasons taskflow directly is a lib. in python, because
, people don't need to learn a new language).

We use python in Ansible all the time:

http://git.openstack.org/cgit/openstack/bifrost/tree/playbooks/library

The reason to use Ansible is that it has already implemented all of
the idempotency and error handling and UI needs that one might need for
running workflows.

I've tried multiple times to understand taskflow, and to me, Ansible is
the anti-taskflow. It's easy to pick up, easy to read the workflows,
doesn't require deep surgery on your code to use (just execute
ansible-playbook), and is full of modules to support nearly anything
your deployment may need.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by Clint_Byrum (40,940 points)   4 6 10
0 votes

Clint Byrum wrote:
Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:

Clint Byrum wrote:

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)
Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?
I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost
Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?

That's an interesting idea. I think a reason Bifrost doesn't just import
nova virt drivers is that they're likely not a supported public API
(despite not having 's at the front). Also, a lot of the reason Bifrost
exists is to enable users to get the benefits of all the baremetal
abstraction work done in Ironic without having to fully embrace all of
OpenStack's core. So while you could get a little bit of the stuff from
nova (like config drive building), you'd still need to handle network
address assignment, image management, etc. etc., and pretty soon you
start having to run a tiny glance and a tiny neutron. The Bifrost way
is the opposite: I just want a tiny Ironic, and _nothing
else.

Ya, I'm just thinking that at a certain point

That seems almost easier than building a bunch of ansible scripts that
appear (at a glance) to do similar things; and u get the benefit of
using a actual programming language vs a
half-programming-ansible-yaml-language...

A realization I'm having is that I'm really not a fan of using ansible
as a half-programming-ansible-yaml-language, which it seems like people
start to try to do after a while (because at some point you need
something like if statements, then things like [1] get created), no
offense to the authors, but I guess this is my personal preference (it's
also one of the reasons taskflow directly is a lib. in python, because
, people don't need to learn a new language).

We use python in Ansible all the time:

http://git.openstack.org/cgit/openstack/bifrost/tree/playbooks/library

The reason to use Ansible is that it has already implemented all of
the idempotency and error handling and UI needs that one might need for
running workflows.

I've tried multiple times to understand taskflow, and to me, Ansible is
the anti-taskflow. It's easy to pick up, easy to read the workflows,
doesn't require deep surgery on your code to use (just execute
ansible-playbook), and is full of modules to support nearly anything
your deployment may need.

Actually they are pretty similar (to a degree), taskflow is pretty much
the same/similar thing ansible is using internally, a graph structure
(last time I checked) that gets ran in parallel or in serial using a
executor concept[1].

Said surgery is only required if you want a deep integration, nothing is
stopping folks from using taskflow in the same manner as running a bunch
of task == similar to ansible style (taskflow also doesn't need to have
its own module concepts as pypi modules primarily just work because it's
python).

But ya, anyway... can't win over everyone ;)

[1] https://github.com/ansible/ansible/tree/devel/lib/ansible/executor


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by harlowja_at_fastmail (16,200 points)   2 6 8
0 votes

Joshua Harlow wrote:
Clint Byrum wrote:

Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:

Clint Byrum wrote:

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created
the following in an attempt to document some of my concerns, and
I'm wondering if you folks could help myself identity ongoing work
to solve these (or alternatives?)
Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?
I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost
Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?

That's an interesting idea. I think a reason Bifrost doesn't just import
nova virt drivers is that they're likely not a supported public API
(despite not having 's at the front). Also, a lot of the reason Bifrost
exists is to enable users to get the benefits of all the baremetal
abstraction work done in Ironic without having to fully embrace all of
OpenStack's core. So while you could get a little bit of the stuff from
nova (like config drive building), you'd still need to handle network
address assignment, image management, etc. etc., and pretty soon you
start having to run a tiny glance and a tiny neutron. The Bifrost way
is the opposite: I just want a tiny Ironic, and _nothing
else.

Ya, I'm just thinking that at a certain point

Oops forgot to fill this out, was just thinking that at a certain point
it might be easier to figure out how to extract that API (meh, if its
public or private) and just have someone make an executive decision
around ironic being a stand-alone thing or not (and a capable
stand-alone thing, not a sorta-standalone-thing).

That seems almost easier than building a bunch of ansible scripts that
appear (at a glance) to do similar things; and u get the benefit of
using a actual programming language vs a
half-programming-ansible-yaml-language...

A realization I'm having is that I'm really not a fan of using ansible
as a half-programming-ansible-yaml-language, which it seems like people
start to try to do after a while (because at some point you need
something like if statements, then things like [1] get created), no
offense to the authors, but I guess this is my personal preference (it's
also one of the reasons taskflow directly is a lib. in python, because
, people don't need to learn a new language).

We use python in Ansible all the time:

http://git.openstack.org/cgit/openstack/bifrost/tree/playbooks/library

The reason to use Ansible is that it has already implemented all of
the idempotency and error handling and UI needs that one might need for
running workflows.

I've tried multiple times to understand taskflow, and to me, Ansible is
the anti-taskflow. It's easy to pick up, easy to read the workflows,
doesn't require deep surgery on your code to use (just execute
ansible-playbook), and is full of modules to support nearly anything
your deployment may need.

Actually they are pretty similar (to a degree), taskflow is pretty much
the same/similar thing ansible is using internally, a graph structure
(last time I checked) that gets ran in parallel or in serial using a
executor concept[1].

Said surgery is only required if you want a deep integration, nothing is
stopping folks from using taskflow in the same manner as running a bunch
of task == similar to ansible style (taskflow also doesn't need to have
its own module concepts as pypi modules primarily just work because it's
python).

But ya, anyway... can't win over everyone ;)

[1] https://github.com/ansible/ansible/tree/devel/lib/ansible/executor


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by harlowja_at_fastmail (16,200 points)   2 6 8
0 votes

On 06/07/2016 09:55 AM, Joshua Harlow wrote:
Joshua Harlow wrote:

Clint Byrum wrote:

Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:

Clint Byrum wrote:

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created
the following in an attempt to document some of my concerns, and
I'm wondering if you folks could help myself identity ongoing work
to solve these (or alternatives?)
Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?
I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost
Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?

That's an interesting idea. I think a reason Bifrost doesn't just import
nova virt drivers is that they're likely not a supported public API
(despite not having 's at the front). Also, a lot of the reason Bifrost
exists is to enable users to get the benefits of all the baremetal
abstraction work done in Ironic without having to fully embrace all of
OpenStack's core. So while you could get a little bit of the stuff from
nova (like config drive building), you'd still need to handle network
address assignment, image management, etc. etc., and pretty soon you
start having to run a tiny glance and a tiny neutron. The Bifrost way
is the opposite: I just want a tiny Ironic, and _nothing
else.

Ya, I'm just thinking that at a certain point

You've got two statements in here, which I'm going to reply to separately:

Oops forgot to fill this out, was just thinking that at a certain point it might
be easier to figure out how to extract that API (meh, if its public or private)

The nova-core team has repeatedly stated that they do not have plans to support
the nova virt driver API as a stable or externally-consumable python API.
Changing that would affect a lot more than Ironic (eg. defcore). A change like
that is not just about what is easier for developers, but also what is better
for the community.

and just have someone make an executive decision around ironic being a
stand-alone thing or not (and a capable stand-alone thing, not a
sorta-standalone-thing).

We already decided to support Ironic as a stand-alone service. So, could you
clarify what you mean when you call it a "sorta-standalone-thing"? In what ways
do you think it's not functional? Do you have specific recommendations on what
we can improve, based on experience using either Ironic or Bifrost?

-Devananda


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by Devananda_van_der_Ve (10,380 points)   2 4 6
0 votes

Devananda van der Veen wrote:
On 06/07/2016 09:55 AM, Joshua Harlow wrote:

Joshua Harlow wrote:

Clint Byrum wrote:

Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:

Clint Byrum wrote:

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:

Hi ironic folks,
As I'm trying to explore how GoDaddy can use ironic I've created
the following in an attempt to document some of my concerns, and
I'm wondering if you folks could help myself identity ongoing work
to solve these (or alternatives?)
Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

List of concerns with ironic:

1.)Nova<-> ironic interactions are generally seem terrible?
I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost
Out of curiosity, why ansible vs turning
https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
(or something like it) into a tiny-wsgi-app (pick useful name here) that
has its own REST api (that looks pretty similar to the public functions
in that driver file)?
That's an interesting idea. I think a reason Bifrost doesn't just import
nova virt drivers is that they're likely not a supported public API
(despite not having 's at the front). Also, a lot of the reason Bifrost
exists is to enable users to get the benefits of all the baremetal
abstraction work done in Ironic without having to fully embrace all of
OpenStack's core. So while you could get a little bit of the stuff from
nova (like config drive building), you'd still need to handle network
address assignment, image management, etc. etc., and pretty soon you
start having to run a tiny glance and a tiny neutron. The Bifrost way
is the opposite: I just want a tiny Ironic, and _nothing
else.

Ya, I'm just thinking that at a certain point

You've got two statements in here, which I'm going to reply to separately:

Oops forgot to fill this out, was just thinking that at a certain point it might
be easier to figure out how to extract that API (meh, if its public or private)

The nova-core team has repeatedly stated that they do not have plans to support
the nova virt driver API as a stable or externally-consumable python API.
Changing that would affect a lot more than Ironic (eg. defcore). A change like
that is not just about what is easier for developers, but also what is better
for the community.

Right, I'm starting to come to the belief that what is better for the
community is to change this; because from what I can tell (from my view
of the world) that tying all the things to nova has really been
detrimental (to a degree) to the whole progression of the cloud as a whole.

It's an opinionated thing to say, yes I understand that, but there
becomes a point where I feel we need to re-evaluate what people really
care about from openstack (because I start to believe that treating the
whole thing as a single product, well that went out the window a long
time ago, with the creation of the big-tent by the TC, with the creation
of mesos, k8s and others by other companies not in openstack...); and
really what's left after that is a bunch of services that to survive (as
a useful set of services) must accept that there is more than just
openstack in the wider world (ie, kubernetes, mesos,
the-next-best-thing...) and if we don't start embracing those other
communities (and no that doesn't mean be an integration engine on-top
or around them) then we are pretty much obsoleting ourselves.

Ya, I know this is a hot (and touchy) topic, and probably other people
don't agree, that's ok... I don't mind the flames, ha (mmmm spicy).

and just have someone make an executive decision around ironic being a
stand-alone thing or not (and a capable stand-alone thing, not a
sorta-standalone-thing).

We already decided to support Ironic as a stand-alone service. So, could you
clarify what you mean when you call it a "sorta-standalone-thing"? In what ways
do you think it's not functional? Do you have specific recommendations on what
we can improve, based on experience using either Ironic or Bifrost?

I'll work on this list, as some folks that are trying start to try to
connect ironic (IMHO without nova, because well kubernetes is enough
like nova that there isn't a need for 2-layers of nova-like-systems at
that point) into kubernetes as a 'resource provider'. I'm sure
attempting to do that will expose a bunch of specific recommendations
soon enough.

-Devananda


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 7, 2016 by harlowja_at_fastmail (16,200 points)   2 6 8
...