settingsLogin | Registersettings

[openstack-dev] [packaging][all] setting minimum version of setuptools in setup.py

0 votes

Similar to pbr, we have a minimum version of setuptools required to
consistently install things in OpenStack. Right now thats 17.1.

However, we don't declare a setup_requires version for it.

I think we should.

setuptools can't self-upgrade, and we don't have declarative deps yet,
so one reaction I expect here is 'how will this help'.

The problem lies in the failure modes. With no dependency declared,
setuptools will try and silently fail, or try and fail with this one
weird error - that doesn't say anything about 'setuptools 3.3. cannot
handle PEP 426 version markers'.

If we set a minimum (but not a maximum) setuptools version as a
setup_requires, I think we'll signal our actual dependencies to
redistributors, and folk consuiming python packages, in a much more
direct fashion. They'll still have to recover manually, but thats ok
IMO. As long as we don't set upper bounds, we won't deadlock ourselves
like we did in the past.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jul 29, 2015 in openstack-dev by Robert_Collins (27,200 points)   4 6 12

5 Responses

0 votes

I agree an error message is better than breaking for insane reasons.

But... maybe as an aside... what about not breaking?

How come the openstack ecosystem doesn't have wait for PEP 426 to be
approved and for setuptools 17.1 to be widely deployed before it can
require/depend on it? Is there no failure/degraded case where we can take
advantage of moving forward in the infrastructure where we apparently need
it - but not necessarily force everyone else to upgrade if it's not
directly solving something for them (e.g. people doing packaging of
openstack projects, but don't personally necessarily currently maintain or
distribute a setuptools package)?

On the web this happens all the time "Sure maybe you can't get the newest
HTML5 wiz-bang, but I can still render something on IE8, OTOH, IE7, wow -
pls upgrade" vs. "How are you not even running setuptools 17.1 right now in
your build environment - it was literally released almost two months
ago!? I... I can't even... it hurts to look at you."

Just Curious.

I only recently found out that PEP 426 was a thing, so I think it's pretty
great to see people driving the python packaging ecosystem forward. For
those involved. Kudos.

-Clay

On Wed, Jul 29, 2015 at 10:27 AM, Robert Collins robertc@robertcollins.net
wrote:

Similar to pbr, we have a minimum version of setuptools required to
consistently install things in OpenStack. Right now thats 17.1.

However, we don't declare a setup_requires version for it.

I think we should.

setuptools can't self-upgrade, and we don't have declarative deps yet,
so one reaction I expect here is 'how will this help'.

The problem lies in the failure modes. With no dependency declared,
setuptools will try and silently fail, or try and fail with this one
weird error - that doesn't say anything about 'setuptools 3.3. cannot
handle PEP 426 version markers'.

If we set a minimum (but not a maximum) setuptools version as a
setup_requires, I think we'll signal our actual dependencies to
redistributors, and folk consuiming python packages, in a much more
direct fashion. They'll still have to recover manually, but thats ok
IMO. As long as we don't set upper bounds, we won't deadlock ourselves
like we did in the past.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 30, 2015 by Clay_Gerrard (5,800 points)   1 2 2
0 votes

On 30 July 2015 at 13:12, Clay Gerrard clay.gerrard@gmail.com wrote:
I agree an error message is better than breaking for insane reasons.

But... maybe as an aside... what about not breaking?

How come the openstack ecosystem doesn't have wait for PEP 426 to be
approved and for setuptools 17.1 to be widely deployed before it can
require/depend on it? Is there no failure/degraded case where we can take
advantage of moving forward in the infrastructure where we apparently need
it - but not necessarily force everyone else to upgrade if it's not directly
solving something for them (e.g. people doing packaging of openstack
projects, but don't personally necessarily currently maintain or distribute
a setuptools package)?

On the web this happens all the time "Sure maybe you can't get the newest
HTML5 wiz-bang, but I can still render something on IE8, OTOH, IE7, wow -
pls upgrade" vs. "How are you not even running setuptools 17.1 right now in
your build environment - it was literally released almost two months
ago!? I... I can't even... it hurts to look at you."

Just Curious.

I only recently found out that PEP 426 was a thing, so I think it's pretty
great to see people driving the python packaging ecosystem forward. For
those involved. Kudos.

So, we [folk in OpenStack that are worried about our packaging] got
involved with upstream pip, setuptools, PEP-426 etc after some years
of trying the 'use whats ready, and workaround the rest by layering on
top' approach.

That approach led us to:
- a nearly diametrically opposite definition of requirements files,
which confuses approximately everyone that reads the pip docs and not
the pbr ones - and vice versa.
- a pattern of emergent bugs where we failed to consider a corner
case because we worked around the surface, which emerges as a bad
behaviour. For instance, our non-compliant version numbers in pbr <
0.10.8 (IIRC the point # correctly). Or the way requirements-py3.txt
files caused use to create universal wheels with bad metadata that
couldn't install properly on other Python releases than the one they
were built on.
- a chunk of overlapping effort, where setuptools and pip were
tackling the same stuff we were, because we're not that special a
snowflake: many folk experience the pains we feel from packaging
limits... but not benefiting from what we'd learnt, nor from the
effort we put in.

So - for OpenStack as a whole, just consuming the current state and
working around was a buggy expensive strategy. And even with that we
still required the latest release of pip, setuptools and virtualenv
simply because we would run into a bug, get it fixed, and need to
consume it in devstack: devstack installs the latest of these four
(add pbr) crucial packaging tools unconditionally and has for a long
time.

So right now I, and a few others, are pushing on:
- Aligning our designs with the ecosystem for less waste and
confusion all around.
- Doing the work we had previously done as layered workarounds as
root level patches to the various tools

While still rolling out the new things, whatever they are, within
OpenStack before we consider them delivered.

HTH,
Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 30, 2015 by Robert_Collins (27,200 points)   4 6 12
0 votes

On 30 July 2015 at 05:27, Robert Collins robertc@robertcollins.net wrote:
Similar to pbr, we have a minimum version of setuptools required to
consistently install things in OpenStack. Right now thats 17.1.

However, we don't declare a setup_requires version for it.

I think we should.

setuptools can't self-upgrade, and we don't have declarative deps yet,
so one reaction I expect here is 'how will this help'.

The problem lies in the failure modes. With no dependency declared,
setuptools will try and silently fail, or try and fail with this one
weird error - that doesn't say anything about 'setuptools 3.3. cannot
handle PEP 426 version markers'.

If we set a minimum (but not a maximum) setuptools version as a
setup_requires, I think we'll signal our actual dependencies to
redistributors, and folk consuiming python packages, in a much more
direct fashion. They'll still have to recover manually, but thats ok
IMO. As long as we don't set upper bounds, we won't deadlock ourselves
like we did in the past.

These are the errors we get with this when the version present is too-old:
Firstly, 0.6c11 which detects the error and reports it.
Then 3.3 which attempts to upgrade setuptools just-in-time but of
course the old setuptools code is what executes, and so the error is
confusing :/. But still better than no hint at all: the presence of
the setuptools upgrade is a signal.

$ pip install setuptools==0.6c11
...
Successfully installed setuptools-0.6rc11
$ pip install .
Processing /home/robertc/work/mock
Complete output from command python setup.py egginfo:
Traceback (most recent call last):
File "", line 20, in
File "/tmp/pip-wnBxi2-build/setup.py", line 6, in
pbr=True)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup
distribution = dist = klass(attrs)
File "/home/robertc/.virtualenvs/test/local/lib/python2.7/site-packages/setuptools/dist.py",
line 260, in init
self.fetchbuildeggs(attrs.pop('setuprequires'))
File "/home/robertc/.virtualenvs/test/local/lib/python2.7/site-packages/setuptools/dist.py",
line 284, in fetch
buildeggs
parse
requirements(requires), installer=self.fetchbuildegg
File "/home/robertc/.virtualenvs/test/local/lib/python2.7/site-packages/pkgresources.py",
line 569, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg
resources.VersionConflict: (setuptools 0.6c11
(/home/robertc/.virtualenvs/test/lib/python2.7/site-packages),
Requirement.parse('setuptools>=17.1'))

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-wnBxi2-build


$ pip install setuptools==3.3
Collecting setuptools==3.3
Downloading setuptools-3.3-py2.py3-none-any.whl (545kB)
100% |████████████████████████████████| 548kB 674kB/s
Installing collected packages: setuptools
Found existing installation: setuptools 18.0.1
Uninstalling setuptools-18.0.1:
Successfully uninstalled setuptools-18.0.1
Successfully installed setuptools-3.3
$ pip install .
Processing /home/robertc/work/mock
Complete output from command python setup.py egg_info:

Installed /tmp/pip-Grkk9a-build/setuptools-18.0.1-py2.7.egg
[pbr] Generating ChangeLog
error in setup command: Invalid environment marker:

(pythonversion<"3.3" and pythonversion>="3")

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-Grkk9a-build

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 31, 2015 by Robert_Collins (27,200 points)   4 6 12
0 votes

Le 29/07/2015 10:27, Robert Collins a écrit :

Similar to pbr, we have a minimum version of setuptools required to
consistently install things in OpenStack. Right now thats 17.1.

However, we don't declare a setup_requires version for it.

I think we should.

If it's possible to explicit the minimum version of setuptools required
to install an application in setup.py, yes, we should do that. I like
being explicit to avoid bad surprises.

Victor


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 17, 2015 by vstinner_at_redhat.c (5,420 points)   1 5 6
0 votes

Le 29/07/2015 18:12, Clay Gerrard a écrit :

I agree an error message is better than breaking for insane reasons.

But... maybe as an aside... what about not breaking?

Well, yes, but who wants to do that?

Old versions of setuptoolsl, pip and pbr have a lot of issues. It will
be a nightmare to workaround them.

For example, it was really difficult to handle dependencies which depend
on the Python version (for python <= 2.6, for python >= 3.0, etc.).
Environment markers solve a real concrete issue. Trying to working
around environment markers is like reinventing the wheel, it doesn't
sound like a good idea to me.

For me, it's a better investment to work upstream on setuptools, pip and
pbr, and require recent versions.

Victor


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 17, 2015 by vstinner_at_redhat.c (5,420 points)   1 5 6
...