settingsLogin | Registersettings

[openstack-dev] [all] Important information for people with in-repo Zuul v3 config

0 votes

Hi everybody,

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again. The status page for Zuulv2 can be found at
http://status.openstack.org/zuul and Zuulv3 can be found at
http://zuulv3.openstack.org.

With the partial rollback of v3, we've left the v3 check pipeline
configured for everyone so that new v3 jobs can be iterated on in
preparation for rolling forward. Doing so leaves open a potential hole
for breakage, so ...

If you propose any changes to your repos that include changes to zuul
config files (.zuul.yaml or .zuul.d) - PLEASE make sure that Zuul v3
runs check jobs and responds before approving the patch.

If you don't don't do that, you could have zuul v2 land a patch that
contains a syntax error that would result in invalid config for v3.
Note that this would break not only your repo - but all testing using
Zuul v3 (in which case we would have to temporarily remove your
repository from the v3 configuration or ask for immediate revert)!

Keep in mind that as we work on diagnosing the issue that caused the
rollback, we could be restarting v3, shutting it down for a bit or it
could be wedged - so v3 might not respond.

Make sure you get a response from v3 on any v3 related patches. Please.

Thanks!
Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 4, 2017 in openstack-dev by Monty_Taylor (22,780 points)   2 5 7

7 Responses

0 votes

On Tue, Oct 3, 2017 at 8:37 PM Monty Taylor mordred@inaugust.com wrote:

Hi everybody,

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again. The status page for Zuulv2 can be found at
http://status.openstack.org/zuul and Zuulv3 can be found at
http://zuulv3.openstack.org.

With the partial rollback of v3, we've left the v3 check pipeline
configured for everyone so that new v3 jobs can be iterated on in
preparation for rolling forward. Doing so leaves open a potential hole
for breakage, so ...

If you propose any changes to your repos that include changes to zuul
config files (.zuul.yaml or .zuul.d) - PLEASE make sure that Zuul v3
runs check jobs and responds before approving the patch.

While work on zuulv3 continues, what's likely to happen rather often is that
patches get a +1 from Jenkins and nothing or some unrelated error from Zuul,
caused perhaps by a restart of zuulv3.

It might be handy to configure a zuulv3-only trigger for patch recheck, to
avoid
consuming the 80% share of zuulv2 nodes to re-run a couple of jobs on zuulv3
side. To that end I proposed [0], I hope you find it useful.

Andrea Frittoli (andreaf)

[0] https://review.openstack.org/#/c/509379/

If you don't don't do that, you could have zuul v2 land a patch that
contains a syntax error that would result in invalid config for v3.
Note that this would break not only your repo - but all testing using
Zuul v3 (in which case we would have to temporarily remove your
repository from the v3 configuration or ask for immediate revert)!

Keep in mind that as we work on diagnosing the issue that caused the
rollback, we could be restarting v3, shutting it down for a bit or it
could be wedged - so v3 might not respond.

Make sure you get a response from v3 on any v3 related patches. Please.

Thanks!
Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On 10/3/17 1:37 PM, Monty Taylor wrote:
The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

[1]
http://logs.openstack.org/01/508901/1/check/gate-tempest-dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.html#_2017-10-04_11_06_07_410466


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Boden_Russell (1,780 points)   2 5
0 votes

On 10/04/2017 07:23 AM, Boden Russell wrote:

On 10/3/17 1:37 PM, Monty Taylor wrote:

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

Yes, this is unrelated.

cffi 1.11.1 was released today on pypi, without wheels. It's installed
super early in the job run because ansible is used to drive things.

There is a stop gap work around that's running tests right now -
https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh

If it passes I'll approve it to get folks back to work. A longer term
solution is probably needed.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Sean_Dague (66,200 points)   4 8 14
0 votes

And we find following error in heat's zuulv2 jobs [1]:

x8664-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D
FORTIFY_SOURCE=2 -g
-fstack-protector-strong -Wformat -Werror=format-security -fPIC
-DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi
-I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o
build/temp.linux-x86_64-2.7/c/_cffi_backend.o

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

[1]
http://logs.openstack.org/27/509227/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/68905a9/console.html#_2017-10-04_08_53_03_706042

2017-10-04 19:23 GMT+08:00 Boden Russell bodenvmw@gmail.com:

On 10/3/17 1:37 PM, Monty Taylor wrote:

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

[1]
http://logs.openstack.org/01/508901/1/check/gate-tempest-
dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.
html#_2017-10-04_11_06_07_410466


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
May The Force of OpenStack Be With You,

Rico Linirc: ricolin


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by rico.lin.guanyu_at_g (1,740 points)   2
0 votes

On Wed, Oct 4, 2017 at 12:23 PM Boden Russell bodenvmw@gmail.com wrote:

On 10/3/17 1:37 PM, Monty Taylor wrote:

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

This is cause by the latest of cffi 1.11.1 which only comes with a tarball
and no wheels as 1.11.0 used to. [0][1][2].
There is a proposed temp d-g patch up to address this [3].

[0]
https://bitbucket.org/cffi/cffi/issues/339/no-wheel-packages-for-1111-version

[1] https://pypi.python.org/pypi/cffi/1.11.0
[2] https://pypi.python.org/pypi/cffi/1.11.1
[3] https://review.openstack.org/#/c/509394/

Andrea Frittoli (andreaf)

[1]

http://logs.openstack.org/01/508901/1/check/gate-tempest-dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.html#_2017-10-04_11_06_07_410466


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On Wed, Oct 4, 2017 at 7:35 AM, Sean Dague sean@dague.net wrote:
On 10/04/2017 07:23 AM, Boden Russell wrote:

On 10/3/17 1:37 PM, Monty Taylor wrote:

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

Yes, this is unrelated.

cffi 1.11.1 was released today on pypi, without wheels. It's installed
super early in the job run because ansible is used to drive things.

There is a stop gap work around that's running tests right now -
https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh

If it passes I'll approve it to get folks back to work. A longer term
solution is probably needed.

might want to consider adding --only-binary <specific_packages|all>
to the pip install line

But it looks like cffi 1.11.1 now has wheels

Jon

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Jon Schlueter
jschluet@redhat.com
IRC: jschlueter/yazug
Senior Software Engineer - OpenStack Productization Engineer


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Jon_Schlueter (240 points)  
0 votes

On Wed, Oct 4, 2017 at 1:00 PM Jon Schlueter jschluet@redhat.com wrote:

On Wed, Oct 4, 2017 at 7:35 AM, Sean Dague sean@dague.net wrote:

On 10/04/2017 07:23 AM, Boden Russell wrote:

On 10/3/17 1:37 PM, Monty Taylor wrote:

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as
your
gate keeper once again.

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/cffibackend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?

Yes, this is unrelated.

cffi 1.11.1 was released today on pypi, without wheels. It's installed
super early in the job run because ansible is used to drive things.

There is a stop gap work around that's running tests right now -
https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh

If it passes I'll approve it to get folks back to work. A longer term
solution is probably needed.

might want to consider adding --only-binary <specific_packages|all>
to the pip install line

Sounds like a good safety net for the future, thanks.

But it looks like cffi 1.11.1 now has wheels

Indeed, thank you

Jon

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Jon Schlueter
jschluet@redhat.com
IRC: jschlueter/yazug
Senior Software Engineer - OpenStack Productization Engineer


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 4, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
...