settingsLogin | Registersettings

[openstack-dev] All patches failing CI for validations stable/ocata branch - help needed

0 votes

Hi all,

currently all branches proposed for the newly created stable/ocata
branch on tripleo-validations are failing CI, and the failures seem to
be unrelated to the content of the patches.[1]

Interestingly, the 3 gates seem to succeed/fail randomly (after a few
rechecks, some gates failed that succeeded before and vice versa). The
error in the logs however is always the same [2] and appears to be
related to an issue with setuptools 34[3], which apparently hit other
openstack projects as well. I tried to create a patch with a fix
(excluding setuptools 34), but couldn't test it because gerrit set the
target patch to master instead of stable/ocata (I guess it only allows
me to propose backport patches to stable branches, not creating a
patch for a stable branch only).

At this point I'm a little out of ideas and would appreciate any help
I can get from folks who are a little more savvy with CI.

Thanks!
Florian

[1] https://review.openstack.org/#/q/status:open+project:openstack/tripleo-validations+branch:stable/ocata
[2] IOError: No such file or directory:
'/home/jenkins/workspace/[...]/pyparsing-2.2.0.dist-info/METADATA'
[3] https://github.com/pypa/setuptools/issues/951


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 30, 2017 in openstack-dev by Florian_Fuchs (520 points)  

4 Responses

0 votes

On Wed, Mar 29, 2017 at 12:32 PM, Florian Fuchs flfuchs@redhat.com wrote:
Hi all,

currently all branches proposed for the newly created stable/ocata
branch on tripleo-validations are failing CI, and the failures seem to
be unrelated to the content of the patches.[1]

Interestingly, the 3 gates seem to succeed/fail randomly (after a few
rechecks, some gates failed that succeeded before and vice versa). The
error in the logs however is always the same [2] and appears to be
related to an issue with setuptools 34[3], which apparently hit other
openstack projects as well. I tried to create a patch with a fix
(excluding setuptools 34), but couldn't test it because gerrit set the
target patch to master instead of stable/ocata (I guess it only allows
me to propose backport patches to stable branches, not creating a
patch for a stable branch only).

At this point I'm a little out of ideas and would appreciate any help
I can get from folks who are a little more savvy with CI.

In the meantime, I suggest we pin setuptools (short term solution, so
we can continue the dev process on this repo).
https://github.com/openstack/openstack-ansible/commit/335a0c8ba1340ac2a1e88e78ae18f426a0447dcc

Thanks!
Florian

[1] https://review.openstack.org/#/q/status:open+project:openstack/tripleo-validations+branch:stable/ocata
[2] IOError: No such file or directory:
'/home/jenkins/workspace/[...]/pyparsing-2.2.0.dist-info/METADATA'
[3] https://github.com/pypa/setuptools/issues/951


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 29, 2017 by emilien_at_redhat.co (36,940 points)   2 6 9
0 votes

On 2017-03-29 18:32:09 +0200 (+0200), Florian Fuchs wrote:
[...]
The error in the logs however is always the same [2] and appears
to be related to an issue with setuptools 34[3], which apparently
hit other openstack projects as well. I tried to create a patch
with a fix (excluding setuptools 34),

The way we "fixed" this for other projects was to update
requirements caps and/or constraints entries for all of setuptools'
direct and transitive install_requires (there are only three, if
memory serves) on stable branches. This is usually not the sort of
thing we'd want to have happening in stable branches, but keeping
them installable with latest setuptools releases is important. Also
pinning setuptools can be hard depending on the situation, since
you're often using setuptools in the process of trying to control
your version of setuptools... bit of a catch-22.

but couldn't test it because gerrit set the target patch to master
instead of stable/ocata (I guess it only allows me to propose
backport patches to stable branches, not creating a patch for a
stable branch only).
[...]

If the .gitreview file in your stable/ocata branch doesn't have
defaultbranch=stable/ocata, then you have to pass the branch name as
a parameter when invoking git review, i.e. git review stable/ocata
(or just fix the .gitreview file so you don't have to think about
it). See https://docs.openstack.org/infra/git-review/usage.html for
details.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 29, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

On Wed, Mar 29, 2017 at 7:20 PM, Emilien Macchi emilien@redhat.com wrote:
On Wed, Mar 29, 2017 at 12:32 PM, Florian Fuchs flfuchs@redhat.com wrote:

Hi all,

currently all branches proposed for the newly created stable/ocata
branch on tripleo-validations are failing CI, and the failures seem to
be unrelated to the content of the patches.[1]

Interestingly, the 3 gates seem to succeed/fail randomly (after a few
rechecks, some gates failed that succeeded before and vice versa). The
error in the logs however is always the same [2] and appears to be
related to an issue with setuptools 34[3], which apparently hit other
openstack projects as well. I tried to create a patch with a fix
(excluding setuptools 34), but couldn't test it because gerrit set the
target patch to master instead of stable/ocata (I guess it only allows
me to propose backport patches to stable branches, not creating a
patch for a stable branch only).

At this point I'm a little out of ideas and would appreciate any help
I can get from folks who are a little more savvy with CI.

In the meantime, I suggest we pin setuptools (short term solution, so
we can continue the dev process on this repo).
https://github.com/openstack/openstack-ansible/commit/335a0c8ba1340ac2a1e88e78ae18f426a0447dcc

Thanks for the quick reply!

Pinning the version directly in tripleo-validations fails as well. It
fixes the initial problem, but now the gate that enforces
compatibility with openstack/requirements/global-requirements.txt
fails. (IIUC, the reason why pinning worked for openstack-ansible is
that the changes there weren't setup.* or *requirement.txt files).

Interestingly, I discovered that openstack/requirements lists a
version of setuptools that addresses the problem (which explains why
we haven't run into that one on tripleo-validations' master branch).
It hasn't been backported to any stable branches though, so I have
proposed one for ocata. If that goes through, it should resolve the
problems in tripleo-validations' ocata branch.

Florian

Thanks!
Florian

[1] https://review.openstack.org/#/q/status:open+project:openstack/tripleo-validations+branch:stable/ocata
[2] IOError: No such file or directory:
'/home/jenkins/workspace/[...]/pyparsing-2.2.0.dist-info/METADATA'
[3] https://github.com/pypa/setuptools/issues/951


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 30, 2017 by Florian_Fuchs (520 points)  
0 votes

On Wed, Mar 29, 2017 at 11:46 PM, Jeremy Stanley fungi@yuggoth.org wrote:
On 2017-03-29 18:32:09 +0200 (+0200), Florian Fuchs wrote:
[...]

The error in the logs however is always the same [2] and appears
to be related to an issue with setuptools 34[3], which apparently
hit other openstack projects as well. I tried to create a patch
with a fix (excluding setuptools 34),

The way we "fixed" this for other projects was to update
requirements caps and/or constraints entries for all of setuptools'
direct and transitive install_requires (there are only three, if
memory serves) on stable branches. This is usually not the sort of
thing we'd want to have happening in stable branches, but keeping
them installable with latest setuptools releases is important. Also
pinning setuptools can be hard depending on the situation, since
you're often using setuptools in the process of trying to control
your version of setuptools... bit of a catch-22.

Thanks for your reply.

FYI: To solve this, I proposed a backport for a requirements patch
that already addresses the problem:
https://review.openstack.org/#/c/451815/

but couldn't test it because gerrit set the target patch to master
instead of stable/ocata (I guess it only allows me to propose
backport patches to stable branches, not creating a patch for a
stable branch only).
[...]

If the .gitreview file in your stable/ocata branch doesn't have
defaultbranch=stable/ocata, then you have to pass the branch name as
a parameter when invoking git review, i.e. git review stable/ocata
(or just fix the .gitreview file so you don't have to think about
it). See https://docs.openstack.org/infra/git-review/usage.html for
details.

We have separate a patch adding the default branch to .gitignore, but,
obviously, it fails due to the problem above...
But thanks for the hint! I wasn't aware the target could be set as a
git review param as well.

Florian

--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 30, 2017 by Florian_Fuchs (520 points)  
...