settingsLogin | Registersettings

[openstack-dev] [ironic] [requirements] moving driver dependencies to global-requirements?

0 votes

Hi all,

So far driver requirements [1] have been managed outside of global-requirements.
This was mostly necessary because some dependencies were not on PyPI. This is no
longer the case, and I'd like to consider managing them just like any other
dependencies. Pros:
1. making these dependencies (and their versions) more visible for packagers
2. following the same policies for regular and driver dependencies
3. ensuring co-installability of these dependencies with each other and with the
remaining openstack
4. potentially using upper-constraints in 3rd party CI to test what packagers
will probably package
5. we'll be able to finally create a tox job running unit tests with all these
dependencies installed (FYI these often breaks in RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams
2. inability to use ironic release notes to explain driver requirements changes
3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will use
setuptools extras to list then in setup.cfg (this way is supported by g-r)
similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

[1] https://github.com/openstack/ironic/blob/master/driver-requirements.txt
[2] https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L115


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 1, 2017 in openstack-dev by Dmitry_Tantsur (18,080 points)   2 3 4

13 Responses

0 votes

Excerpts from Dmitry Tantsur's message of 2017-10-30 17:51:49 +0100:

Hi all,

So far driver requirements [1] have been managed outside of global-requirements.
This was mostly necessary because some dependencies were not on PyPI. This is no
longer the case, and I'd like to consider managing them just like any other
dependencies. Pros:
1. making these dependencies (and their versions) more visible for packagers
2. following the same policies for regular and driver dependencies
3. ensuring co-installability of these dependencies with each other and with the
remaining openstack
4. potentially using upper-constraints in 3rd party CI to test what packagers
will probably package
5. we'll be able to finally create a tox job running unit tests with all these
dependencies installed (FYI these often breaks in RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams
2. inability to use ironic release notes to explain driver requirements changes
3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will use
setuptools extras to list then in setup.cfg (this way is supported by g-r)
similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

The second option is what I would expect.

Doug

[1] https://github.com/openstack/ironic/blob/master/driver-requirements.txt
[2] https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L115


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 30, 2017 by Doug_Hellmann (87,520 points)   3 4 4
0 votes

The second seem to be better suited for per driver requirement handling and per HW type per function.
Which option is easier to handle for container per dependency for the future?

Thanks,
Arkady

-----Original Message-----
From: Doug Hellmann [mailto:doug@doughellmann.com]
Sent: Monday, October 30, 2017 2:47 PM
To: openstack-dev openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [ironic] [requirements] moving driver dependencies to global-requirements?

Excerpts from Dmitry Tantsur's message of 2017-10-30 17:51:49 +0100:
Hi all,

So far driver requirements [1] have been managed outside of global-requirements.
This was mostly necessary because some dependencies were not on PyPI.
This is no longer the case, and I'd like to consider managing them
just like any other dependencies. Pros:
1. making these dependencies (and their versions) more visible for
packagers 2. following the same policies for regular and driver
dependencies 3. ensuring co-installability of these dependencies with
each other and with the remaining openstack 4. potentially using
upper-constraints in 3rd party CI to test what packagers will probably
package 5. we'll be able to finally create a tox job running unit
tests with all these dependencies installed (FYI these often breaks in
RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams 2.
inability to use ironic release notes to explain driver requirements
changes 3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will
use setuptools extras to list then in setup.cfg (this way is supported
by g-r) similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

The second option is what I would expect.

Doug

[1]
https://github.com/openstack/ironic/blob/master/driver-requirements.tx
t [2]
https://github.com/openstack/python-ironicclient/blob/master/setup.cfg

L115


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 30, 2017 by Arkady.Kanevsky_at_d (1,480 points)   1
0 votes

On 17-10-30 20:48:37, Arkady.Kanevsky@dell.com wrote:
The second seem to be better suited for per driver requirement handling and per HW type per function.
Which option is easier to handle for container per dependency for the future?

Thanks,
Arkady

-----Original Message-----
From: Doug Hellmann [mailto:doug@doughellmann.com]
Sent: Monday, October 30, 2017 2:47 PM
To: openstack-dev openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [ironic] [requirements] moving driver dependencies to global-requirements?

Excerpts from Dmitry Tantsur's message of 2017-10-30 17:51:49 +0100:

Hi all,

So far driver requirements [1] have been managed outside of global-requirements.
This was mostly necessary because some dependencies were not on PyPI.
This is no longer the case, and I'd like to consider managing them
just like any other dependencies. Pros:
1. making these dependencies (and their versions) more visible for
packagers 2. following the same policies for regular and driver
dependencies 3. ensuring co-installability of these dependencies with
each other and with the remaining openstack 4. potentially using
upper-constraints in 3rd party CI to test what packagers will probably
package 5. we'll be able to finally create a tox job running unit
tests with all these dependencies installed (FYI these often breaks in
RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams 2.
inability to use ironic release notes to explain driver requirements
changes 3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will
use setuptools extras to list then in setup.cfg (this way is supported
by g-r) similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

The second option is what I would expect.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Meant to reply from this address, but below is my original response.

The first question I have is if ALL the drivers are suposed to be co-installable
with eachother. If so, adding them to requirements sounds fine, as long as each
one follows https://github.com/openstack/requirements/#for-new-requirements .

As far as the format, I prefer option 2 (the breakout option). I'm not sure if
the bot will need an update, but I suspect not as it tries to keep ordering iirc.

--
Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Oct 30, 2017 by prometheanfire_at_ge (6,880 points)   1 3 5
0 votes

From: Dmitry Tantsur [mailto:dtantsur@redhat.com]

Cons:
1. more work for both the requirements team and the vendor teams

Please elaborate on the additional work you envision for the vendor teams.

  1. inability to use ironic release notes to explain driver requirements changes

Where could that information move to?

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

Overall, a big +1. I prefer the second approach.

A couple of questions ...

  1. If two (2) hardware types have the same requirement, would they both
    enter it in their lists?
  2. And would that be correctly handled?

    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 30, 2017 by Richard.Pioso_at_del (300 points)  
0 votes

Excerpts from Richard.Pioso's message of 2017-10-30 23:11:31 +0000:

From: Dmitry Tantsur [mailto:dtantsur@redhat.com]

Cons:
1. more work for both the requirements team and the vendor teams

Please elaborate on the additional work you envision for the vendor teams.

  1. inability to use ironic release notes to explain driver requirements changes

Where could that information move to?

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

Overall, a big +1. I prefer the second approach.

A couple of questions ...

  1. If two (2) hardware types have the same requirement, would they both
    enter it in their lists?

Yes.

  1. And would that be correctly handled?

Good question. We should test the requirements update script to see.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 31, 2017 by Doug_Hellmann (87,520 points)   3 4 4
0 votes

On Mon, Oct 30, 2017 at 9:46 PM, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Dmitry Tantsur's message of 2017-10-30 17:51:49 +0100:

Hi all,

So far driver requirements [1] have been managed outside of
global-requirements.
This was mostly necessary because some dependencies were not on PyPI.
This is no
longer the case, and I'd like to consider managing them just like any
other
dependencies. Pros:
1. making these dependencies (and their versions) more visible for
packagers
2. following the same policies for regular and driver dependencies
3. ensuring co-installability of these dependencies with each other and
with the
remaining openstack
4. potentially using upper-constraints in 3rd party CI to test what
packagers
will probably package
5. we'll be able to finally create a tox job running unit tests with all
these
dependencies installed (FYI these often breaks in RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams
2. inability to use ironic release notes to explain driver requirements
changes
3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will use
setuptools extras to list then in setup.cfg (this way is supported by
g-r)
similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

The second option is what I would expect.

I also would prefer option 2 (per driver extras), with a catch-all [all]
extra to install all of those in one short command

I have a separate concern about ansible-deploy interface. It obviously
depends on Ansibe, but so does most of upstream infra. I am not sure if
adding (somehow restricted) Ansible to g-r won't break anything.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 31, 2017 by Pavlo_Shchelokovskyy (4,760 points)   3 4
0 votes

On 17:51 Oct 30, Dmitry Tantsur wrote:
Hi all,

So far driver requirements [1] have been managed outside of
global-requirements. This was mostly necessary because some dependencies
were not on PyPI. This is no longer the case, and I'd like to consider
managing them just like any other dependencies. Pros:
1. making these dependencies (and their versions) more visible for packagers
2. following the same policies for regular and driver dependencies
3. ensuring co-installability of these dependencies with each other and with
the remaining openstack
4. potentially using upper-constraints in 3rd party CI to test what
packagers will probably package
5. we'll be able to finally create a tox job running unit tests with all
these dependencies installed (FYI these often breaks in RDO CI)

I noticed Cinder is doing this with drivers, but in a different file:

http://git.openstack.org/cgit/openstack/cinder/tree/driver-requirements.txt

--
Mike Perez


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Oct 31, 2017 by Mike_Perez (13,120 points)   2 3 4
0 votes

On Mon, Oct 30, 2017 at 05:51:49PM +0100, Dmitry Tantsur wrote:
Hi all,

So far driver requirements [1] have been managed outside of
global-requirements. This was mostly necessary because some dependencies
were not on PyPI. This is no longer the case, and I'd like to consider
managing them just like any other dependencies. Pros:
1. making these dependencies (and their versions) more visible for packagers
2. following the same policies for regular and driver dependencies
3. ensuring co-installability of these dependencies with each other and with
the remaining openstack
4. potentially using upper-constraints in 3rd party CI to test what
packagers will probably package
5. we'll be able to finally create a tox job running unit tests with all
these dependencies installed (FYI these often breaks in RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams

The biggest impact on the requirements team is the initial review to
validate that the driver library meets all the criteria [1]

There are 11 requirements in driver-requirements.txt 8 would be new.

I haven't done a review but there is a non-zero chance one of these 8
could be rejected. At that point you're half in/out and that leaves
ironic in an awkward place.

  1. inability to use ironic release notes to explain driver requirements changes

You can still do this, it's just harder because you don't have a single
point where the updates are happening. At the moment when a change is
proposed to modify driver-requirements you can reject the change until
it has a release note. Now you'd need to check at release time if a
release note was added.

We can probably come up with a compromise solution. Perhaps annotate
global requirements like:


sushy is used by ironic-drivers check for release note

sushy>=0.1.0


Then in theory the requirements team could defer the change until it
depends on a merged release note in ironic? It isn't 100% human proof.

I haven't really thought about it though ...

  1. any objections from the requirements team?

No objection from me but see the points above.

If we make this change, we'll drop driver-requirements.txt, and will use
setuptools extras to list then in setup.cfg (this way is supported by g-r)
similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

This one please.

Yours Tony.

[1] http://git.openstack.org/cgit/openstack/requirements/tree/README.rst#n265


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 1, 2017 by Tony_Breeds (19,660 points)   2 6 6
0 votes

On Mon, Oct 30, 2017 at 08:07:10PM -0400, Doug Hellmann wrote:
Excerpts from Richard.Pioso's message of 2017-10-30 23:11:31 +0000:

  1. And would that be correctly handled?

Good question. We should test the requirements update script to see.

It does. I did a quick fictional test:

This:


diff --git a/setup.cfg b/setup.cfg
index 6c3242535..01469c39f 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -25,6 +25,11 @@ packages =
ironic
ironictempestplugin

+[extras]
+redfish1 =
+ sushy>=0.0.9 # Apache-2.0
+redfish2 =
+ sushy>=0.0.10 # Apache-2.0
[entrypoints]
oslo.config.opts =
ironic = ironic.conf.opts:list
opts


Becomes:

[extras]
redfish1 =
- sushy>=0.0.9 # Apache-2.0
+ sushy>=0.1.0 # Apache-2.0
redfish2 =
- sushy>=0.0.10 # Apache-2.0
+ sushy>=0.1.0 # Apache-2.0


After running the bot

Yours Tony.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 1, 2017 by Tony_Breeds (19,660 points)   2 6 6
0 votes

On 10/30/2017 11:28 PM, Matthew Thode wrote:
On 17-10-30 20:48:37, Arkady.Kanevsky@dell.com wrote:

The second seem to be better suited for per driver requirement handling and per HW type per function.
Which option is easier to handle for container per dependency for the future?

Thanks,
Arkady

-----Original Message-----
From: Doug Hellmann [mailto:doug@doughellmann.com]
Sent: Monday, October 30, 2017 2:47 PM
To: openstack-dev openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [ironic] [requirements] moving driver dependencies to global-requirements?

Excerpts from Dmitry Tantsur's message of 2017-10-30 17:51:49 +0100:

Hi all,

So far driver requirements [1] have been managed outside of global-requirements.
This was mostly necessary because some dependencies were not on PyPI.
This is no longer the case, and I'd like to consider managing them
just like any other dependencies. Pros:
1. making these dependencies (and their versions) more visible for
packagers 2. following the same policies for regular and driver
dependencies 3. ensuring co-installability of these dependencies with
each other and with the remaining openstack 4. potentially using
upper-constraints in 3rd party CI to test what packagers will probably
package 5. we'll be able to finally create a tox job running unit
tests with all these dependencies installed (FYI these often breaks in
RDO CI)

Cons:
1. more work for both the requirements team and the vendor teams 2.
inability to use ironic release notes to explain driver requirements
changes 3. any objections from the requirements team?

If we make this change, we'll drop driver-requirements.txt, and will
use setuptools extras to list then in setup.cfg (this way is supported
by g-r) similar to what we do in ironicclient [2].

We either will have one list:

[extras]
drivers =
sushy>=a.b
python-dracclient>=x.y
python-prolianutils>=v.w
...

or (and I like this more) we'll have a list per hardware type:

[extras]
redfish =
sushy>=a.b
idrac =
python-dracclient>=x.y
ilo =
...
...

WDYT?

The second option is what I would expect.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Meant to reply from this address, but below is my original response.

The first question I have is if ALL the drivers are suposed to be co-installable
with eachother. If so, adding them to requirements sounds fine, as long as each
one follows https://github.com/openstack/requirements/#for-new-requirements .

Yes, an ironic installation can have all drivers enabled at the same time on the
same conductor.

As far as the format, I prefer option 2 (the breakout option). I'm not sure if
the bot will need an update, but I suspect not as it tries to keep ordering iirc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 15, 2017 by Dmitry_Tantsur (18,080 points)   2 3 4
...