settingsLogin | Registersettings

[openstack-dev] [requirements] global-requirements not co-installable

0 votes

I don't know if they are intended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

conflict because openstack-doc-tools has no releases >=0.23
compatible with oslo.config>=1.11.0:
e.g.
Cached requires for ('openstack-doc-tools', (), <Version('0.26.0')>)
(('argparse', (), <SpecifierSet('')>), ('oslo.config', (),
<SpecifierSet('<1.10.0,>=1.9.3')>), ('pbr', (),
<SpecifierSet('!=0.7,<1.0,>=0.6')>), ('iso8601', (),
<SpecifierSet('>=0.1.9')>), ('lxml', (), <SpecifierSet('>=2.3')>),
('demjson', (), <SpecifierSet('')>), ('babel', (),
<SpecifierSet('>=1.3')>), ('sphinx', (),
<SpecifierSet('!=1.2.0,!=1.3b1,<1.3,>=1.1.2')>))

(this is output from the debug trace in my pip resolver branch).

Now, this is probably never showing up in the gate today, but
something as simple as:
pip install oslo.config
pip install 'openstack-doc-tools>=0.23'
will show the issue:
$ pip install oslo.config
Downloading/unpacking oslo.config
Downloading oslo.config-1.11.0-py2.py3-none-any.whl (67kB): 67kB downloaded
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0
in /home/robertc/.virtualenvs/pip-test/lib/python2.7/site-packages
(from oslo.config)
Downloading/unpacking pbr>=0.6,!=0.7,<1.0 (from oslo.config)
Downloading pbr-0.11.0-py2.py3-none-any.whl (78kB): 78kB downloaded
Downloading/unpacking stevedore>=1.3.0 (from oslo.config)
Downloading stevedore-1.4.0-py2.py3-none-any.whl
Downloading/unpacking netaddr>=0.7.12 (from oslo.config)
Downloading netaddr-0.7.14-py2.py3-none-any.whl (1.5MB): 1.5MB downloaded
Requirement already satisfied (use --upgrade to upgrade): argparse in
/usr/lib/python2.7 (from oslo.config)
Requirement already satisfied (use --upgrade to upgrade): pip in
/home/robertc/.virtualenvs/pip-test/lib/python2.7/site-packages (from
pbr>=0.6,!=0.7,<1.0->oslo.config)
Installing collected packages: oslo.config, pbr, stevedore, netaddr
Successfully installed oslo.config pbr stevedore netaddr
Cleaning up...
$ pip install 'openstack-doc-tools>=0.23'
Downloading/unpacking openstack-doc-tools>=0.23
Downloading openstackdoctools-0.26.0-py2.py3-none-any.whl (180kB):
180kB downloaded
Downloading/unpacking iso8601>=0.1.9 (from openstack-doc-tools>=0.23)
Downloading iso8601-0.1.10.tar.gz
Running setup.py
(path:/home/robertc/.virtualenvs/pip-test/build/iso8601/setup.py)
egg_info for package iso8601
Downloading/unpacking oslo.config>=1.9.3,<1.10.0 (from
openstack-doc-tools>=0.23)
Downloading oslo.config-1.9.3-py2.py3-none-any.whl (67kB): 67kB downloaded
^^^^^^^^^^^
....

So this will downgrade oslo.config to meet the requirements - and that
could of course be very very surprising to the gate ;)

I don't have a fully formed view yet on complete co-installability in
our global requirements. But without that it may be hard to calculate
a working set that we can pin (if we go all-pinned), and its certainly
harder to write tooling to do any sort of analysis because we have to
start partitioning it into sets that work together, and those that
don't.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 8, 2015 in openstack-dev by Robert_Collins (27,200 points)   4 6 12

19 Responses

0 votes

On 05/08/2015 10:02 AM, Robert Collins wrote:
I don't know if they are intended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

We haven't imported yet the new requirements for oslo.config and
released a new version of openstack-doc-tools. I'll take care of this,

Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu,
Graham Norton, HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Andreas_Jaeger (17,140 points)   2 3 4
0 votes

On 05/08/2015 10:02 AM, Robert Collins wrote:
I don't know if they areintended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

These are not the only culprits, grepping requirements in repositories
in the openstack namespace, I see:

castellan/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
ceilometermiddleware/requirements.txt:oslo.config>=1.9.3,<1.10.0 #
Apache-2.0
congress/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
glance/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
gnocchi/requirements-py3.txt:oslo.config<1.10.0,>=1.9.3
gnocchi/requirements.txt:oslo.config<1.10.0,>=1.9.3
ironic-lib/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
ironic-python-agent/requirements.txt:oslo.config>=1.9.3,<1.10.0 #
Apache-2.0
magnum/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
murano-agent/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
tempest/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
zaqar/requirements-py3.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
zaqar/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0

Note that some are not in requirements/projects.txt.

Still, openstack-doc-tools should be fixed later today,
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu,
Graham Norton, HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Andreas_Jaeger (17,140 points)   2 3 4
0 votes

On 8 May 2015 at 20:39, Andreas Jaeger aj@suse.com wrote:
On 05/08/2015 10:02 AM, Robert Collins wrote:

I don't know if they areintended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

These are not the only culprits, grepping requirements in repositories in
the openstack namespace, I see:

The key issue will be the intersection with entries in global-requirements.txt:

ceilometermiddleware/requirements.txt:oslo.config>=1.9.3,<1.10.0 #

is the only one - the rest aren't in global-requirements at all.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Robert_Collins (27,200 points)   4 6 12
0 votes

On 05/08/2015 10:12 AM, Andreas Jaeger wrote:
On 05/08/2015 10:02 AM, Robert Collins wrote:

I don't know if they are intended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

We haven't imported yet the new requirements for oslo.config and
released a new version of openstack-doc-tools. I'll take care of this,

Fixed now - openstack-doc-tools 0.27 got released,

Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu,
Graham Norton, HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Andreas_Jaeger (17,140 points)   2 3 4
0 votes

I'm slightly confused how we got there, because we do try to install
everything all at once in the test jobs -
http://logs.openstack.org/83/181083/1/check/check-requirements-integration-dsvm/4effcf7/console.html#_2015-05-07_17_49_26_699

And it seemed to work, you can find similar lines in previous changes as
well. That was specifically added as a check for these kinds of issues.
Is this a race in the resolution?

On 05/08/2015 04:02 AM, Robert Collins wrote:
I don't know if they are intended to be, but right now there is no
set of versions that can be co-installed, of everything listed in
global requirements.

I don't have a full set of the conflicts (because I don't have a good
automatic trace for 'why X is unresolvable' - its nontrivial).

However right now:
openstack-doc-tools>=0.23
and
oslo.config>=1.11.0

conflict because openstack-doc-tools has no releases >=0.23
compatible with oslo.config>=1.11.0:
e.g.
Cached requires for ('openstack-doc-tools', (), <Version('0.26.0')>)
(('argparse', (), <SpecifierSet('')>), ('oslo.config', (),
<SpecifierSet('<1.10.0,>=1.9.3')>), ('pbr', (),
<SpecifierSet('!=0.7,<1.0,>=0.6')>), ('iso8601', (),
<SpecifierSet('>=0.1.9')>), ('lxml', (), <SpecifierSet('>=2.3')>),
('demjson', (), <SpecifierSet('')>), ('babel', (),
<SpecifierSet('>=1.3')>), ('sphinx', (),
<SpecifierSet('!=1.2.0,!=1.3b1,<1.3,>=1.1.2')>))

(this is output from the debug trace in my pip resolver branch).

Now, this is probably never showing up in the gate today, but
something as simple as:
pip install oslo.config
pip install 'openstack-doc-tools>=0.23'
will show the issue:
$ pip install oslo.config
Downloading/unpacking oslo.config
Downloading oslo.config-1.11.0-py2.py3-none-any.whl (67kB): 67kB downloaded
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0
in /home/robertc/.virtualenvs/pip-test/lib/python2.7/site-packages
(from oslo.config)
Downloading/unpacking pbr>=0.6,!=0.7,<1.0 (from oslo.config)
Downloading pbr-0.11.0-py2.py3-none-any.whl (78kB): 78kB downloaded
Downloading/unpacking stevedore>=1.3.0 (from oslo.config)
Downloading stevedore-1.4.0-py2.py3-none-any.whl
Downloading/unpacking netaddr>=0.7.12 (from oslo.config)
Downloading netaddr-0.7.14-py2.py3-none-any.whl (1.5MB): 1.5MB downloaded
Requirement already satisfied (use --upgrade to upgrade): argparse in
/usr/lib/python2.7 (from oslo.config)
Requirement already satisfied (use --upgrade to upgrade): pip in
/home/robertc/.virtualenvs/pip-test/lib/python2.7/site-packages (from
pbr>=0.6,!=0.7,<1.0->oslo.config)
Installing collected packages: oslo.config, pbr, stevedore, netaddr
Successfully installed oslo.config pbr stevedore netaddr
Cleaning up...
$ pip install 'openstack-doc-tools>=0.23'
Downloading/unpacking openstack-doc-tools>=0.23
Downloading openstackdoctools-0.26.0-py2.py3-none-any.whl (180kB):
180kB downloaded
Downloading/unpacking iso8601>=0.1.9 (from openstack-doc-tools>=0.23)
Downloading iso8601-0.1.10.tar.gz
Running setup.py
(path:/home/robertc/.virtualenvs/pip-test/build/iso8601/setup.py)
egg_info for package iso8601
Downloading/unpacking oslo.config>=1.9.3,<1.10.0 (from
openstack-doc-tools>=0.23)
Downloading oslo.config-1.9.3-py2.py3-none-any.whl (67kB): 67kB downloaded
^^^^^^^^^^^
....

So this will downgrade oslo.config to meet the requirements - and that
could of course be very very surprising to the gate ;)

I don't have a fully formed view yet on complete co-installability in
our global requirements. But without that it may be hard to calculate
a working set that we can pin (if we go all-pinned), and its certainly
harder to write tooling to do any sort of analysis because we have to
start partitioning it into sets that work together, and those that
don't.

-Rob

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Sean_Dague (66,200 points)   4 10 16
0 votes

On 8 May 2015 at 22:54, Sean Dague sean@dague.net wrote:
I'm slightly confused how we got there, because we do try to install
everything all at once in the test jobs -
http://logs.openstack.org/83/181083/1/check/check-requirements-integration-dsvm/4effcf7/console.html#_2015-05-07_17_49_26_699

And it seemed to work, you can find similar lines in previous changes as
well. That was specifically added as a check for these kinds of issues.
Is this a race in the resolution?

What resolution :).

So what happens with pip install -r
/opt/stack/new/requirements/global-requirements.txt is that the
constraints in that file are all immediately put into pip's state,
including oslo.config >= 1.11.0, and then all other constraints that
reference to oslo.config are simply ignored. this is 1b (and 2a) on
https://github.com/pypa/pip/issues/988.

IOW we haven't been testing what we've thought we've been testing.
What we've been testing is that 'python setup.py install X' for X in
global-requirements.txt works, which sadly doesn't tell us a lot at
all.

So, as I have a working (but unpolished) resolver, when I try to do
the same thing, it chews away at the problem and concludes that no, it
can't do it - because its no longer ignoring the additional
constraints.

To get out of the hole, we might consider using pip-compile now as a
warning job - if it can succeed we'll be able to be reasonably
confident that pip itself will succeed once the resolver is merged.

The resolver I have doesn't preserve the '1b' feature at all at this
point, and we're going to need to find a way to separate out 'I want
X' from 'I want X and I know better than you', which will let folk get
into tasty tasty trouble (like we're in now).

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Robert_Collins (27,200 points)   4 6 12
0 votes

On 05/08/2015 07:13 AM, Robert Collins wrote:
On 8 May 2015 at 22:54, Sean Dague sean@dague.net wrote:

I'm slightly confused how we got there, because we do try to install
everything all at once in the test jobs -
http://logs.openstack.org/83/181083/1/check/check-requirements-integration-dsvm/4effcf7/console.html#_2015-05-07_17_49_26_699

And it seemed to work, you can find similar lines in previous changes as
well. That was specifically added as a check for these kinds of issues.
Is this a race in the resolution?

What resolution :).

So what happens with pip install -r
/opt/stack/new/requirements/global-requirements.txt is that the
constraints in that file are all immediately put into pip's state,
including oslo.config >= 1.11.0, and then all other constraints that
reference to oslo.config are simply ignored. this is 1b (and 2a) on
https://github.com/pypa/pip/issues/988.

IOW we haven't been testing what we've thought we've been testing.
What we've been testing is that 'python setup.py install X' for X in
global-requirements.txt works, which sadly doesn't tell us a lot at
all.

So, as I have a working (but unpolished) resolver, when I try to do
the same thing, it chews away at the problem and concludes that no, it
can't do it - because its no longer ignoring the additional
constraints.

To get out of the hole, we might consider using pip-compile now as a
warning job - if it can succeed we'll be able to be reasonably
confident that pip itself will succeed once the resolver is merged.

The resolver I have doesn't preserve the '1b' feature at all at this
point, and we're going to need to find a way to separate out 'I want
X' from 'I want X and I know better than you', which will let folk get
into tasty tasty trouble (like we're in now).

Gotcha, so, yes, so the subtleties of pip were lost here.

Instead of using another tool, could we make a version of this job pull
and use the prerelease version of your pip code. Then we can run the
same tests and fix them in a non voting job against this code that has
not yet released.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Sean_Dague (66,200 points)   4 10 16
0 votes

On Fri, May 8, 2015 at 4:23 AM, Sean Dague sean@dague.net wrote:

On 05/08/2015 07:13 AM, Robert Collins wrote:

On 8 May 2015 at 22:54, Sean Dague sean@dague.net wrote:

I'm slightly confused how we got there, because we do try to install
everything all at once in the test jobs -

http://logs.openstack.org/83/181083/1/check/check-requirements-integration-dsvm/4effcf7/console.html#_2015-05-07_17_49_26_699

And it seemed to work, you can find similar lines in previous changes as
well. That was specifically added as a check for these kinds of issues.
Is this a race in the resolution?

What resolution :).

So what happens with pip install -r
/opt/stack/new/requirements/global-requirements.txt is that the
constraints in that file are all immediately put into pip's state,
including oslo.config >= 1.11.0, and then all other constraints that
reference to oslo.config are simply ignored. this is 1b (and 2a) on
https://github.com/pypa/pip/issues/988.

IOW we haven't been testing what we've thought we've been testing.
What we've been testing is that 'python setup.py install X' for X in
global-requirements.txt works, which sadly doesn't tell us a lot at
all.

So, as I have a working (but unpolished) resolver, when I try to do
the same thing, it chews away at the problem and concludes that no, it
can't do it - because its no longer ignoring the additional
constraints.

To get out of the hole, we might consider using pip-compile now as a
warning job - if it can succeed we'll be able to be reasonably
confident that pip itself will succeed once the resolver is merged.

The resolver I have doesn't preserve the '1b' feature at all at this
point, and we're going to need to find a way to separate out 'I want
X' from 'I want X and I know better than you', which will let folk get
into tasty tasty trouble (like we're in now).

Gotcha, so, yes, so the subtleties of pip were lost here.

Instead of using another tool, could we make a version of this job pull
and use the prerelease version of your pip code. Then we can run the
same tests and fix them in a non voting job against this code that has
not yet released.

Once we are actually testing that all of global requirements is
co-installable will we end up with even more cases like this? Or is this
just an artifact of capping fro kilo?
https://review.openstack.org/#/c/166377/

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Joe_Gordon (24,620 points)   2 5 8
0 votes

On 9 May 2015 at 05:51, Joe Gordon joe.gordon0@gmail.com wrote:

Once we are actually testing that all of global requirements is
co-installable will we end up with even more cases like this? Or is this
just an artifact of capping fro kilo?
https://review.openstack.org/#/c/166377/

As I read it, we've got some tooling that isn't PEP-440 compatible
(https://www.python.org/dev/peps/pep-0440/#compatible-release defines
~=) and as such we had to rollback the intended use of that. As long
as we identify and fix those tools, we should be fine. Did anyone
involved with that situation create a bug we can use to track this? I
don't think it has anything to do with the choice of cap-or-not.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Robert_Collins (27,200 points)   4 6 12
0 votes

On 8 May 2015 at 23:23, Sean Dague sean@dague.net wrote:
On 05/08/2015 07:13 AM, Robert Collins wrote:

The resolver I have doesn't preserve the '1b' feature at all at this
point, and we're going to need to find a way to separate out 'I want
X' from 'I want X and I know better than you', which will let folk get
into tasty tasty trouble (like we're in now).

Gotcha, so, yes, so the subtleties of pip were lost here.

Instead of using another tool, could we make a version of this job pull
and use the prerelease version of your pip code. Then we can run the
same tests and fix them in a non voting job against this code that has
not yet released.

Thats certainly possible too. Upside: if it works we know it works in
pip. Downside, we'll be tracking something that is in active
development and late-prototype /early alpha stage. This is likely to
be in pip 8.0 (7.0 should be out inside of a week).

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 8, 2015 by Robert_Collins (27,200 points)   4 6 12
...