settingsLogin | Registersettings

[openstack-dev] Fwd: [Distutils][pbr] Announcement: Pip 10 is coming, and will move all internal APIs

0 votes

It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

--- Begin forwarded message from Paul Moore ---
From: Paul Moore p.f.moore@gmail.com
To: Distutils distutils-sig@python.org, pypa-dev pypa-dev@googlegroups.com
Date: Fri, 20 Oct 2017 14:22:03 +0100
Subject: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

We're in the process of starting to plan for a release of pip (the
long-awaited pip 10). We're likely still a month or two away from a
release, but now is the time for people to start ensuring that
everything works for them. One key change in the new version will be
that all of the internal APIs of pip will no longer be available, so
any code that currently calls functions in the "pip" namespace will
break. Calling pip's internal APIs has never been supported, and
always carried a risk of such breakage, so projects doing so should,
in theory, be prepared for such things. However, reality is not always
that simple, and we are aware that people will need time to deal with
the implications.

Just in case it's not clear, simply finding where the internal APIs
have moved to and calling them under the new names is not what
people should do. We can't stop people calling the internal APIs,
obviously, but the idea of this change is to give people the incentive
to find a supported approach, not just to annoy people who are doing
things we don't want them to ;-)

So please - if you're calling pip's internals in your code, take the
opportunity now to check out the in-development version of pip, and
ensure your project will still work when pip 10 is released.

And many thanks to anyone else who helps by testing out the new
version, as well :-)

Thanks,
Paul
--- End forwarded message ---


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 21, 2017 in openstack-dev by Doug_Hellmann (87,520 points)   3 4 9

8 Responses

0 votes

On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:
It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

After a quick skim of PBR I don't think we use pip internals anywhere.
Its all executed via the command itself. That said we should test this
so I've put up https://review.openstack.org/513825 (others should feel
free to iterate on it if it doesn't work) to install latest pip master
in a devstack run.

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On Fri, Oct 20, 2017, at 11:17 AM, Clark Boylan wrote:
On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:

It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

After a quick skim of PBR I don't think we use pip internals anywhere.
Its all executed via the command itself. That said we should test this
so I've put up https://review.openstack.org/513825 (others should feel
free to iterate on it if it doesn't work) to install latest pip master
in a devstack run.

And it has already caught its first bug. Fix and details at
https://review.openstack.org/#/c/513832/.

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On Fri, Oct 20, 2017, at 11:17 AM, Clark Boylan wrote:
On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:

It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

After a quick skim of PBR I don't think we use pip internals anywhere.
Its all executed via the command itself. That said we should test this
so I've put up https://review.openstack.org/513825 (others should feel
free to iterate on it if it doesn't work) to install latest pip master
in a devstack run.

The current issue this change is facing can be seen at
http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
The tl;dr is that for distutils installed packages (basically all the
distro installed python packges) pip refuses to uninstall them in order
to perform upgrades because it can't reliably determine where all the
files are. I think this is a new pip 10 behavior.

In the general case I think this means we can not rely on global pip
installs anymore. This may be a good thing to bring up with upstream
PyPA as I expect it will break a lot of people in a lot of places (it
will break infra for example too).

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Clark_Boylan (8,800 points)   1 2 4
0 votes

Excerpts from Clark Boylan's message of 2017-10-20 13:14:13 -0700:

On Fri, Oct 20, 2017, at 11:17 AM, Clark Boylan wrote:

On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:

It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

After a quick skim of PBR I don't think we use pip internals anywhere.
Its all executed via the command itself. That said we should test this
so I've put up https://review.openstack.org/513825 (others should feel
free to iterate on it if it doesn't work) to install latest pip master
in a devstack run.

The current issue this change is facing can be seen at
http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
The tl;dr is that for distutils installed packages (basically all the
distro installed python packges) pip refuses to uninstall them in order
to perform upgrades because it can't reliably determine where all the
files are. I think this is a new pip 10 behavior.

In the general case I think this means we can not rely on global pip
installs anymore. This may be a good thing to bring up with upstream
PyPA as I expect it will break a lot of people in a lot of places (it
will break infra for example too).

Clark

Yes, it would be good for someone who understands the cases where we're
doing these sorts of global package installations using pip to post on
distutils-sig explaining the breakage and asking for some time to let us
work around the issue.

We have a couple of ways to mitigate it, depending on the situation.

  1. Pin the affected packages to the system package versions in our
    constraints list.

  2. Install into a virtualenv that ignores system packages, avoiding the
    need to upgrade at all.

  3. Remove the system package before installing from pip (why is it there
    in the first place?).

It's not clear to me which is the best approach. I've added
[devstack][qa] to the subject line to draw some more attention to
this thread from folks familiar with these jobs.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Doug_Hellmann (87,520 points)   3 4 9
0 votes

On Fri, Oct 20, 2017 at 4:32 PM, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Clark Boylan's message of 2017-10-20 13:14:13 -0700:

On Fri, Oct 20, 2017, at 11:17 AM, Clark Boylan wrote:

On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:

It sounds like the PyPI/PyPA folks are planning some major changes to
pip internals, soon.

I know pbr uses setuptools, and I don't think it uses pip, but if
someone has time to verify that it would be helpful.

We'll also want to watch out for breakage in normal use of pip 10. If
they're making changes this big, they may miss something in their own
test coverage that affects our jobs.

After a quick skim of PBR I don't think we use pip internals anywhere.
Its all executed via the command itself. That said we should test this
so I've put up https://review.openstack.org/513825 (others should feel
free to iterate on it if it doesn't work) to install latest pip master
in a devstack run.

The current issue this change is facing can be seen at
http://logs.openstack.org/25/513825/4/check/legacy-tempest-
dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
The tl;dr is that for distutils installed packages (basically all the
distro installed python packges) pip refuses to uninstall them in order
to perform upgrades because it can't reliably determine where all the
files are. I think this is a new pip 10 behavior.

In the general case I think this means we can not rely on global pip
installs anymore. This may be a good thing to bring up with upstream
PyPA as I expect it will break a lot of people in a lot of places (it
will break infra for example too).

Clark

Yes, it would be good for someone who understands the cases where we're
doing these sorts of global package installations using pip to post on
distutils-sig explaining the breakage and asking for some time to let us
work around the issue.

We have a couple of ways to mitigate it, depending on the situation.

  1. Pin the affected packages to the system package versions in our
    constraints list.

  2. Install into a virtualenv that ignores system packages, avoiding the
    need to upgrade at all.

  3. Remove the system package before installing from pip (why is it there
    in the first place?).

It's not clear to me which is the best approach. I've added
[devstack][qa] to the subject line to draw some more attention to
this thread from folks familiar with these jobs.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I have used option 2 with great success to avoid the issue of system
package conflicts for a couple of years now, back when pip would still
uninstall system packages for upgrades.

Option 3 is difficult to make work since so many packages have dependancies
on python-* packages, I abandoned that approach fairly quickly.

Thanks,
SamYaple


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Sam_Yaple (3,560 points)   3 3
0 votes

On 2017-10-20 17:50:15 -0400 (-0400), Sam Yaple wrote:
On Fri, Oct 20, 2017 at 4:32 PM, Doug Hellmann doug@doughellmann.com wrote:
[...]

  1. Remove the system package before installing from pip (why is
    it there in the first place?).
    [...]
    Option 3 is difficult to make work since so many packages have
    dependancies on python-* packages, I abandoned that approach
    fairly quickly.

Right, on Debian/Ubuntu it's not too terrible (cloud-init's
dependencies are usually the biggest issue there and we manage to
avoid them by building our own images with no cloud-init), but on
Red Hat derivatives there are a lot of deep operating system
internals built on top of packaged Python libraries which simply
can't be uninstalled cleanly nor safely.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Oct 21, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

On 10/21/2017 07:14 AM, Clark Boylan wrote:
The current issue this change is facing can be seen at
http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
The tl;dr is that for distutils installed packages (basically all the
distro installed python packges) pip refuses to uninstall them in order
to perform upgrades because it can't reliably determine where all the
files are. I think this is a new pip 10 behavior.

In the general case I think this means we can not rely on global pip
installs anymore. This may be a good thing to bring up with upstream
PyPA as I expect it will break a lot of people in a lot of places (it
will break infra for example too).

deja-vu! pip 8 tried this and quickly reverted. I wrote a long email
with all the details, but then figured that's not going to help much
so translated it into [1].

-i

[1] https://github.com/pypa/pip/issues/4805


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 23, 2017 by Ian_Wienand (3,620 points)   4 5
0 votes

On 10/22/2017 12:18 AM, Jeremy Stanley wrote:
Right, on Debian/Ubuntu it's not too terrible (cloud-init's
dependencies are usually the biggest issue there and we manage to
avoid them by building our own images with no cloud-init), but on
Red Hat derivatives there are a lot of deep operating system
internals built on top of packaged Python libraries which simply
can't be uninstalled cleanly nor safely.

Also note though, if it can be uninstalled, we have often had problems
with the packages coming back and overwriting the pip installed
version, which leads to often very obscure problems. For this reason
in various bits of devstack/devstack-gate/dib's pip install etc we
often install and pin packages to let pip overwrite them.

-i


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 23, 2017 by Ian_Wienand (3,620 points)   4 5
...