settingsLogin | Registersettings

[openstack-dev] [tripleo] Please do not approve or recheck anything not related to CI alert bugs

0 votes

Hey Folks

So we're at 24+ hours again in the gate[0] and the queue only
continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
approve of recheck anything that isn't related to these bugs. I will
most likely need to go through the queue and abandon everything to
clear it up as we are consistently hitting timeouts on various jobs
which is preventing anything from merging.

Thanks,
-Alex

[0] http://zuulv3.openstack.org/
[1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+alert&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 15, 2017 in openstack-dev by aschultz_at_redhat.c (5,800 points)   2 2 2

4 Responses

0 votes

So we have some good news and some bad news. The good news is that
we've managed to get the gate queue[0] under control since we've held
off on pushing new things to the gate. The bad news is that we've
still got some random failures occurring during the deployment of
master. Since we're not seeing infra related issues, we should be OK
to merge things to stable/* branches. Unfortunately until we resolve
the issues in master[1] we could potentially backup the queue. Please
do not merge things that are not critical bugs. I would ask that
folks please take a look at the open bugs and help figure out what is
going wrong. I've created two issues today that I've seen in the gate
that we don't appear to have open patches for. One appears to be an
issue in the heat deployment process[3] and the other is related to
the tempest verification of being able to launch a VM & ssh to it[4].

Thanks,
-Alex

[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731063

On Tue, Nov 7, 2017 at 8:33 AM, Alex Schultz aschultz@redhat.com wrote:
Hey Folks

So we're at 24+ hours again in the gate[0] and the queue only
continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
approve of recheck anything that isn't related to these bugs. I will
most likely need to go through the queue and abandon everything to
clear it up as we are consistently hitting timeouts on various jobs
which is preventing anything from merging.

Thanks,
-Alex

[0] http://zuulv3.openstack.org/
[1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+alert&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 2
0 votes

Ok so here's the current status of things. I've gone through some of
the pending patches and sent them to the gate over the weekend since
the gate was empty (yay!). We've managed to land a bunch of patches.
That being said for any patch for master with scenario jobs, please do
not recheck/approve. Currently the non-containerized scenario001/004
jobs are broken due to Bug 1731688[0] (these run on
tripleo-quickstart-extras/tripleo-ci). There is a patch[1] out for a
revert of the breaking change. The scenario001-container job is super
flaky due to Bug 1731063[2] and we could use some help figuring out
what's going on. We're also seeing some issues around heat
interactions[3][4] but those seems to be less of a problem than the
previously mentioned bugs.

So at the moment any changes that don't have scenario jobs associated
with them may be approved/rechecked freely. We can discuss on Monday
what to do about the scenario jobs if we still are running into issues
without a solution in sight. Also please keep an eye on the gate
queue[5] and don't approve things if it starts getting excessively
long.

Thanks,
-Alex

[0] https://bugs.launchpad.net/tripleo/+bug/1731688
[1] https://review.openstack.org/#/c/519041/
[2] https://bugs.launchpad.net/tripleo/+bug/1731063
[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731540
[5] http://zuulv3.openstack.org/

On Wed, Nov 8, 2017 at 3:39 PM, Alex Schultz aschultz@redhat.com wrote:
So we have some good news and some bad news. The good news is that
we've managed to get the gate queue[0] under control since we've held
off on pushing new things to the gate. The bad news is that we've
still got some random failures occurring during the deployment of
master. Since we're not seeing infra related issues, we should be OK
to merge things to stable/* branches. Unfortunately until we resolve
the issues in master[1] we could potentially backup the queue. Please
do not merge things that are not critical bugs. I would ask that
folks please take a look at the open bugs and help figure out what is
going wrong. I've created two issues today that I've seen in the gate
that we don't appear to have open patches for. One appears to be an
issue in the heat deployment process[3] and the other is related to
the tempest verification of being able to launch a VM & ssh to it[4].

Thanks,
-Alex

[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731063

On Tue, Nov 7, 2017 at 8:33 AM, Alex Schultz aschultz@redhat.com wrote:

Hey Folks

So we're at 24+ hours again in the gate[0] and the queue only
continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
approve of recheck anything that isn't related to these bugs. I will
most likely need to go through the queue and abandon everything to
clear it up as we are consistently hitting timeouts on various jobs
which is preventing anything from merging.

Thanks,
-Alex

[0] http://zuulv3.openstack.org/
[1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+alert&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 12, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 2
0 votes

On Sat, Nov 11, 2017 at 10:47 PM, Alex Schultz aschultz@redhat.com wrote:

Ok so here's the current status of things. I've gone through some of
the pending patches and sent them to the gate over the weekend since
the gate was empty (yay!). We've managed to land a bunch of patches.
That being said for any patch for master with scenario jobs, please do
not recheck/approve. Currently the non-containerized scenario001/004
jobs are broken due to Bug 1731688[0] (these run on
tripleo-quickstart-extras/tripleo-ci). There is a patch[1] out for a
revert of the breaking change. The scenario001-container job is super
flaky due to Bug 1731063[2] and we could use some help figuring out
what's going on. We're also seeing some issues around heat
interactions[3][4] but those seems to be less of a problem than the
previously mentioned bugs.

So at the moment any changes that don't have scenario jobs associated
with them may be approved/rechecked freely. We can discuss on Monday
what to do about the scenario jobs if we still are running into issues
without a solution in sight. Also please keep an eye on the gate
queue[5] and don't approve things if it starts getting excessively
long.

Thanks,
-Alex

[0] https://bugs.launchpad.net/tripleo/+bug/1731688
[1] https://review.openstack.org/#/c/519041/
[2] https://bugs.launchpad.net/tripleo/+bug/1731063
[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731540
[5] http://zuulv3.openstack.org/

On Wed, Nov 8, 2017 at 3:39 PM, Alex Schultz aschultz@redhat.com wrote:

So we have some good news and some bad news. The good news is that
we've managed to get the gate queue[0] under control since we've held
off on pushing new things to the gate. The bad news is that we've
still got some random failures occurring during the deployment of
master. Since we're not seeing infra related issues, we should be OK
to merge things to stable/* branches. Unfortunately until we resolve
the issues in master[1] we could potentially backup the queue. Please
do not merge things that are not critical bugs. I would ask that
folks please take a look at the open bugs and help figure out what is
going wrong. I've created two issues today that I've seen in the gate
that we don't appear to have open patches for. One appears to be an
issue in the heat deployment process[3] and the other is related to
the tempest verification of being able to launch a VM & ssh to it[4].

Thanks,
-Alex

[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731063

On Tue, Nov 7, 2017 at 8:33 AM, Alex Schultz aschultz@redhat.com
wrote:

Hey Folks

So we're at 24+ hours again in the gate[0] and the queue only
continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
approve of recheck anything that isn't related to these bugs. I will
most likely need to go through the queue and abandon everything to
clear it up as we are consistently hitting timeouts on various jobs
which is preventing anything from merging.

Thanks,
-Alex

[0] http://zuulv3.openstack.org/
[1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-
importance&field.status%3Alist=NEW&field.status%
3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%
3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&
assigneeoption=any&field.assignee=&field.bugreporter=&
field.bugcommenter=&field.subscriber=&field.structural
subscriber=&field.tag=ci+alert&field.tagscombinator=
ALL&field.has
cve.used=&field.omitdupes.used=&field.omit
dupes=on&field.affectsme.used=&field.haspatch.used=&
field.hasbranches.used=&field.hasbranches=on&field.
hasnobranches.used=&field.hasnobranches=on&field.has_
blueprints.used=&field.hasblueprints=on&field.hasno_
blueprints.used=&field.hasnoblueprints=on&search=Search


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks for continuing to push on this Alex!


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 13, 2017 by Wesley_Hayutin (2,320 points)   2
0 votes

Ok so here's the latest. We've switched scenario001 to non-voting[0]
for now until Bug 1731063[1] can be resolved. We should be OK to start
merging thing in master as the other current issues don't appear to be
affecting the gate significantly as it stands. We still need to
understand why we're hitting Bug 1731063 and address the problem so we
can revert the non-voting change ASAP. Scenario001 provides lots of
coverage for TripleO so I do not want to see it non-voting for long.
If scenario001 is failing on your change, please make sure it is not
Bug 1731063 before rechecking or approving. If you are approving
changes or rechecking and it fails, do not blindly recheck. Please
file a new bug and ping #tripleo so we can make sure we don't have
other things that may affect the gate.

Thanks,
-Alex

[0] https://review.openstack.org/#/c/520155/
[1] https://bugs.launchpad.net/tripleo/+bug/1731063

On Sat, Nov 11, 2017 at 8:47 PM, Alex Schultz aschultz@redhat.com wrote:
Ok so here's the current status of things. I've gone through some of
the pending patches and sent them to the gate over the weekend since
the gate was empty (yay!). We've managed to land a bunch of patches.
That being said for any patch for master with scenario jobs, please do
not recheck/approve. Currently the non-containerized scenario001/004
jobs are broken due to Bug 1731688[0] (these run on
tripleo-quickstart-extras/tripleo-ci). There is a patch[1] out for a
revert of the breaking change. The scenario001-container job is super
flaky due to Bug 1731063[2] and we could use some help figuring out
what's going on. We're also seeing some issues around heat
interactions[3][4] but those seems to be less of a problem than the
previously mentioned bugs.

So at the moment any changes that don't have scenario jobs associated
with them may be approved/rechecked freely. We can discuss on Monday
what to do about the scenario jobs if we still are running into issues
without a solution in sight. Also please keep an eye on the gate
queue[5] and don't approve things if it starts getting excessively
long.

Thanks,
-Alex

[0] https://bugs.launchpad.net/tripleo/+bug/1731688
[1] https://review.openstack.org/#/c/519041/
[2] https://bugs.launchpad.net/tripleo/+bug/1731063
[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731540
[5] http://zuulv3.openstack.org/

On Wed, Nov 8, 2017 at 3:39 PM, Alex Schultz aschultz@redhat.com wrote:

So we have some good news and some bad news. The good news is that
we've managed to get the gate queue[0] under control since we've held
off on pushing new things to the gate. The bad news is that we've
still got some random failures occurring during the deployment of
master. Since we're not seeing infra related issues, we should be OK
to merge things to stable/* branches. Unfortunately until we resolve
the issues in master[1] we could potentially backup the queue. Please
do not merge things that are not critical bugs. I would ask that
folks please take a look at the open bugs and help figure out what is
going wrong. I've created two issues today that I've seen in the gate
that we don't appear to have open patches for. One appears to be an
issue in the heat deployment process[3] and the other is related to
the tempest verification of being able to launch a VM & ssh to it[4].

Thanks,
-Alex

[3] https://bugs.launchpad.net/tripleo/+bug/1731032
[4] https://bugs.launchpad.net/tripleo/+bug/1731063

On Tue, Nov 7, 2017 at 8:33 AM, Alex Schultz aschultz@redhat.com wrote:

Hey Folks

So we're at 24+ hours again in the gate[0] and the queue only
continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
approve of recheck anything that isn't related to these bugs. I will
most likely need to go through the queue and abandon everything to
clear it up as we are consistently hitting timeouts on various jobs
which is preventing anything from merging.

Thanks,
-Alex

[0] http://zuulv3.openstack.org/
[1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+alert&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 15, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 2
...