settingsLogin | Registersettings

[openstack-dev] [all] New setuptools release, and the world is broken

0 votes

FYI if you see the following error in your job logs, you have the new setuptools to thank:

AttributeError: Distribution instance has no attribute 'install_requires'

I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the issue, and very quickly got a response and there’s a PR up to resolve it.

In our case we saw this when downgrading setuptools to our known, good working version.

I’d like to suggest that we include setuptools, pip, wheel and other core packages in the upper constraints management process and that all the images built for jobs make use of it. The number of times that a new release of pip/setuptools has completely ground development to a halt for a day, sometimes more, is a little too frequent for my liking.

IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to ‘pip freeze –all’ for the output to include the versions for pip, setuptools and wheel. Then, with that spec, pip can be installed using u-c like so:

CURLCMD="curl --silent --show-error --retry 5"
OUTPUT
FILE="get-pip.py"
${CURLCMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUTFILE} ||\
${CURLCMD} https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > ${OUTPUTFILE}

python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt

That will ensure a consistent, known good version set is installed and will also cater for the situation where the primary URL for get-pip.py is down (as happens sometimes).


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jul 14, 2017 in openstack-dev by Jesse.Pretorius_at_r (2,260 points)   1 2

9 Responses

0 votes

Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +0000:

FYI if you see the following error in your job logs, you have the new setuptools to thank:

AttributeError: Distribution instance has no attribute 'install_requires'

I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the issue, and very quickly got a response and there’s a PR up to resolve it.

In our case we saw this when downgrading setuptools to our known, good working version.

I’d like to suggest that we include setuptools, pip, wheel and other core packages in the upper constraints management process and that all the images built for jobs make use of it. The number of times that a new release of pip/setuptools has completely ground development to a halt for a day, sometimes more, is a little too frequent for my liking.

IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to ‘pip freeze –all’ for the output to include the versions for pip, setuptools and wheel. Then, with that spec, pip can be installed using u-c like so:

CURLCMD="curl --silent --show-error --retry 5"
OUTPUT
FILE="get-pip.py"
${CURLCMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUTFILE} ||\
${CURLCMD} https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > ${OUTPUTFILE}

python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt

That will ensure a consistent, known good version set is installed and will also cater for the situation where the primary URL for get-pip.py is down (as happens sometimes).

I know we made the explicit decision not to pin setuptools, but I don't
remember the motivation. Was it a technical decision (we can't) or
because it seemed like a bad idea (we want to ensure we have the
latest)?

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Doug_Hellmann (87,520 points)   3 4 8
0 votes

On 2017-07-14 10:50:50 -0400 (-0400), Doug Hellmann wrote:
Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +0000:

FYI if you see the following error in your job logs, you have the new setuptools to thank:

AttributeError: Distribution instance has no attribute 'install_requires'

I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the issue, and very quickly got a response and there’s a PR up to resolve it.

In our case we saw this when downgrading setuptools to our known, good working version.

I’d like to suggest that we include setuptools, pip, wheel and other core packages in the upper constraints management process and that all the images built for jobs make use of it. The number of times that a new release of pip/setuptools has completely ground development to a halt for a day, sometimes more, is a little too frequent for my liking.

IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to ‘pip freeze –all’ for the output to include the versions for pip, setuptools and wheel. Then, with that spec, pip can be installed using u-c like so:

CURLCMD="curl --silent --show-error --retry 5"
OUTPUT
FILE="get-pip.py"
${CURLCMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUTFILE} ||\
${CURLCMD} https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > ${OUTPUTFILE}

python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt

That will ensure a consistent, known good version set is installed and will also cater for the situation where the primary URL for get-pip.py is down (as happens sometimes).

I know we made the explicit decision not to pin setuptools, but I don't
remember the motivation. Was it a technical decision (we can't) or
because it seemed like a bad idea (we want to ensure we have the
latest)?

Chicken and egg. Once you get to the point where pip can enforce
constraints, you already have a version of setuptools installed. And
as evidenced by, for example, this current bug you would just end up
breaking on the downgrade trying to replace your existing broken
version with whatever version is requested. Also you would need a
separate phase to upgrade/downgrade setuptools separate from other
packages using it.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 14, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

Excerpts from Jeremy Stanley's message of 2017-07-14 15:17:52 +0000:

On 2017-07-14 10:50:50 -0400 (-0400), Doug Hellmann wrote:

Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +0000:

FYI if you see the following error in your job logs, you have the new setuptools to thank:

AttributeError: Distribution instance has no attribute 'install_requires'

I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the issue, and very quickly got a response and there’s a PR up to resolve it.

In our case we saw this when downgrading setuptools to our known, good working version.

I’d like to suggest that we include setuptools, pip, wheel and other core packages in the upper constraints management process and that all the images built for jobs make use of it. The number of times that a new release of pip/setuptools has completely ground development to a halt for a day, sometimes more, is a little too frequent for my liking.

IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to ‘pip freeze –all’ for the output to include the versions for pip, setuptools and wheel. Then, with that spec, pip can be installed using u-c like so:

CURLCMD="curl --silent --show-error --retry 5"
OUTPUT
FILE="get-pip.py"
${CURLCMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUTFILE} ||\
${CURLCMD} https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > ${OUTPUTFILE}

python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt

That will ensure a consistent, known good version set is installed and will also cater for the situation where the primary URL for get-pip.py is down (as happens sometimes).

I know we made the explicit decision not to pin setuptools, but I don't
remember the motivation. Was it a technical decision (we can't) or
because it seemed like a bad idea (we want to ensure we have the
latest)?

Chicken and egg. Once you get to the point where pip can enforce
constraints, you already have a version of setuptools installed. And
as evidenced by, for example, this current bug you would just end up
breaking on the downgrade trying to replace your existing broken
version with whatever version is requested. Also you would need a
separate phase to upgrade/downgrade setuptools separate from other
packages using it.

That makes sense.

I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Doug_Hellmann (87,520 points)   3 4 8
0 votes

On 7/14/17, 4:17 PM, "Jeremy Stanley" fungi@yuggoth.org wrote:

Chicken and egg. Once you get to the point where pip can enforce
constraints, you already have a version of setuptools installed. And
as evidenced by, for example, this current bug you would just end up
breaking on the downgrade trying to replace your existing broken
version with whatever version is requested. Also you would need a
separate phase to upgrade/downgrade setuptools separate from other
packages using it.

Which is why I think we should do it right at the point where we put pip there. This can be done with the get-pip script which only requires python to be there.

I did a little spelunking to try and figure out where it is we put pip onto the nodepool nodes. It appears that d-i-b for Ubuntu Xenial uses packages to put it there, but that doesn’t explain how the latest version got there. Does anyone have any insight into where exactly the latest version is installed?


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

On 7/14/17, 4:54 PM, "Doug Hellmann" doug@doughellmann.com wrote:

I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.

It has been capable of this for quite some time. You can provide both requirements
And constraints.

python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0

Or, far better, is to use constraints because it’ll ensure that any dependent packages are also the right versions.

python /opt/get-pip.py pip setuptools wheel –constraint http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

On 2017-07-14 16:05:36 +0000 (+0000), Jesse Pretorius wrote:
On 7/14/17, 4:54 PM, "Doug Hellmann" doug@doughellmann.com wrote:
[...]

I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.

It has been capable of this for quite some time. You can provide
both requirements And constraints.

python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0

Or, far better, is to use constraints because it’ll ensure that
any dependent packages are also the right versions.

python /opt/get-pip.py pip setuptools wheel –constraint
http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt

Is there a mechanism to leverage this in tox or when invoking
virtualenv? We don't run get-pip.py in most jobs because our images
have pip/setuptools preinstalled to get around bootstrapping issues,
though I suppose that could with some effort be moved into job
runtime as a (very early) builder macro. Using constraints to
control these during image generation doesn't make a whole lot of
sense though as images are only rebuilt once a day and so tracking
these in the constraints list won't be self-testing in that regard
anyway.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 14, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

On 2017-07-14 15:55:40 +0000 (+0000), Jesse Pretorius wrote:
I did a little spelunking to try and figure out where it is we put
pip onto the nodepool nodes. It appears that d-i-b for Ubuntu
Xenial uses packages to put it there, but that doesn’t explain how
the latest version got there. Does anyone have any insight into
where exactly the latest version is installed?

This has changed slightly since we excised the remaining Puppet
setup from our images, so it's no longer done by the puppet-pip
module. It seems the pip-and-virtualenv element is currently
bringing that in.

--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jul 14, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

On 7/14/17, 5:18 PM, "Jeremy Stanley" fungi@yuggoth.org wrote:

Is there a mechanism to leverage this in tox or when invoking
virtualenv? We don't run get-pip.py in most jobs because our images
have pip/setuptools preinstalled to get around bootstrapping issues,
though I suppose that could with some effort be moved into job
runtime as a (very early) builder macro. Using constraints to
control these during image generation doesn't make a whole lot of
sense though as images are only rebuilt once a day and so tracking
these in the constraints list won't be self-testing in that regard
anyway.

Fair enough, although that’s what we have today so I don’t see it as worse. In fact, I think it’s an improvement because our constraints can be updated when we choose to (through the normal requirements management process) rather than whenever the new package releases.

I think it may make sense to do a pre-tox macro which preps pip/setuptools/wheel using get-pip.py, then places the relevant wheels into the virtualenv_support directory [1] so that tox uses them when building virtualenvs. Another option could be to use a configuration file [2] to specify a directory to find the right wheels in.

[1] https://virtualenv.pypa.io/en/stable/userguide/#the-extra-search-dir-option
[2] https://virtualenv.pypa.io/en/stable/reference/#configuration-file


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

Excerpts from Jeremy Stanley's message of 2017-07-14 16:18:04 +0000:

On 2017-07-14 16:05:36 +0000 (+0000), Jesse Pretorius wrote:

On 7/14/17, 4:54 PM, "Doug Hellmann" doug@doughellmann.com wrote:
[...]

I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.

It has been capable of this for quite some time. You can provide
both requirements And constraints.

python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0

Or, far better, is to use constraints because it’ll ensure that
any dependent packages are also the right versions.

python /opt/get-pip.py pip setuptools wheel –constraint
http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt

Is there a mechanism to leverage this in tox or when invoking
virtualenv? We don't run get-pip.py in most jobs because our images
have pip/setuptools preinstalled to get around bootstrapping issues,
though I suppose that could with some effort be moved into job
runtime as a (very early) builder macro. Using constraints to
control these during image generation doesn't make a whole lot of
sense though as images are only rebuilt once a day and so tracking
these in the constraints list won't be self-testing in that regard
anyway.

I was thinking we would use an early stage builder to do it, too.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jul 14, 2017 by Doug_Hellmann (87,520 points)   3 4 8
...