settingsLogin | Registersettings

[openstack-dev] [all] 3rd Party CI vs. Gerrit

0 votes

It's clear that lots of projects want 3rd Party CI information on
patches. But it's also clear that 6 months into this experiment with a
lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

A couple of things have fallen out of this. 3rd Party CI bots outnumber
Human comments on changes on some projects (Nova / Neutron). That has an
impact on the readability of the votes section (on a neutron change the
files in the change are rarely above the fold), the readability of the
comments.

3rd Party CI systems haven't become all that reliable. They fall into
the same problems that Jenkins hits with cloud networking, race bugs in
OpenStack, but also new bugs around site configs. It's kind of a
testament to how much we've learned about how to keep the machine
running that the upstream CI system, even with all it's faults, still
trends more reliable than most of the 3rd Party systems.

Commenting in Gerrit was to eventually get towards voting in Gerrit. But
my experience at this point is reviewers are at CI fatigue and are
mostly not paying attention to the votes. Heck, when we're dealing with
a bunch of bugs in the gate most reviewers want to ignore the Jenkins
votes too, which is why you get the recheck grinding behavior.

This has all gone far enough that someone actually wrote a Grease Monkey
script to purge all the 3rd Party CI content out of Jenkins UI. People
are writing mail filters to dump all the notifications. Dan Berange
filters all them out of his gerrit query tools.

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

What if 3rd Party CI didn't vote in Gerrit? What if it instead published
to some 3rd party test reporting site (a thing that doesn't yet exist).
Gerrit has the facility so that we could inject the dashboard content
for this in Gerrit in a little table somewhere, but the data would
fundamentally live outside of Gerrit. It would also mean that all the
aggregate reporting of 3rd Party CI that's being done in custom gerrit
scripts, could be integrated directly into such a thing.

I'm not signing up for this particular mission, but I wanted to stick it
out there to figure out if the idea had merrit, and if it did, if it
excited anyone to enough to dive on it.

-Sean

--
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20140627/043342bc/attachment.pgp

asked Jun 27, 2014 in openstack-dev by Sean_Dague (66,200 points)   4 8 14
retagged Feb 25, 2015 by admin

20 Responses

0 votes

On 27/06/14 07:40 -0400, Sean Dague wrote:
It's clear that lots of projects want 3rd Party CI information on
patches. But it's also clear that 6 months into this experiment with a
lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

A couple of things have fallen out of this. 3rd Party CI bots outnumber
Human comments on changes on some projects (Nova / Neutron). That has an
impact on the readability of the votes section (on a neutron change the
files in the change are rarely above the fold), the readability of the
comments.

3rd Party CI systems haven't become all that reliable. They fall into
the same problems that Jenkins hits with cloud networking, race bugs in
OpenStack, but also new bugs around site configs. It's kind of a
testament to how much we've learned about how to keep the machine
running that the upstream CI system, even with all it's faults, still
trends more reliable than most of the 3rd Party systems.

Commenting in Gerrit was to eventually get towards voting in Gerrit. But
my experience at this point is reviewers are at CI fatigue and are
mostly not paying attention to the votes. Heck, when we're dealing with
a bunch of bugs in the gate most reviewers want to ignore the Jenkins
votes too, which is why you get the recheck grinding behavior.

This has all gone far enough that someone actually wrote a Grease Monkey
script to purge all the 3rd Party CI content out of Jenkins UI. People
are writing mail filters to dump all the notifications. Dan Berange
filters all them out of his gerrit query tools.

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

What if 3rd Party CI didn't vote in Gerrit? What if it instead published
to some 3rd party test reporting site (a thing that doesn't yet exist).
Gerrit has the facility so that we could inject the dashboard content
for this in Gerrit in a little table somewhere, but the data would
fundamentally live outside of Gerrit. It would also mean that all the
aggregate reporting of 3rd Party CI that's being done in custom gerrit
scripts, could be integrated directly into such a thing.

I'm not signing up for this particular mission, but I wanted to stick it
out there to figure out if the idea had merrit, and if it did, if it
excited anyone to enough to dive on it.

I agree all those comments cause way to much noise. The most important
bit of the comment is the link that takes you to the CI logs so, it
may be probably enough to just have 1 link that will take someone to
the builds of the 3rd party CI for that specific change.

Along the lines of what you just proposed, what if each 3rd Party CI
username in the review's list just links to this "builds-list"
resources that a developer can go to to check the logs. Would that be
too much of a hack for gerrit?

I'd like to avoid us to create yet-another-monitoring-resource that
we'll have to go to in order to figure out what's going on.

FWIW, the above could also be done for our Jenkins.

Overall, +1 for the email and idea.
Flavio

--
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL:

responded Jun 27, 2014 by Flavio_Percoco (36,960 points)   3 7 11
0 votes

On Fri, Jun 27, 2014 at 07:40:51AM -0400, Sean Dague wrote:
It's clear that lots of projects want 3rd Party CI information on
patches. But it's also clear that 6 months into this experiment with a
lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

That's an understatement about the UI :-)

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

What if 3rd Party CI didn't vote in Gerrit? What if it instead published
to some 3rd party test reporting site (a thing that doesn't yet exist).
Gerrit has the facility so that we could inject the dashboard content
for this in Gerrit in a little table somewhere, but the data would
fundamentally live outside of Gerrit. It would also mean that all the
aggregate reporting of 3rd Party CI that's being done in custom gerrit
scripts, could be integrated directly into such a thing.

Agreed, it would be a great improvement in usability if we stopped all
CI systems, including our default Jenkins, from ever commenting on
reviews. At most gating CIs should +1/-1. Having a table of results
displayed, pulling the data from an external result tracking system
would be a great idea.

Even better if this external system had a nice button you can press
to trigger re-check, so we can stop using comments for that too.

To me the ideal world is where the only things adding comments to
reviews are human and their comments are actually about the code
in the patch :-)

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

responded Jun 27, 2014 by Daniel_P._Berrange (27,920 points)   2 4 9
0 votes

On Fri, Jun 27, 2014 at 6:35 AM, Daniel P. Berrange
wrote:

On Fri, Jun 27, 2014 at 07:40:51AM -0400, Sean Dague wrote:

It's clear that lots of projects want 3rd Party CI information on
patches. But it's also clear that 6 months into this experiment with a
lot of 3rd Party CI systems, the Gerrit UI is really not great for this.

That's an understatement about the UI :-)

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

?Yes please, and no I'm not signing up for it either :)
?

What if 3rd Party CI didn't vote in Gerrit? What if it instead published
to some 3rd party test reporting site (a thing that doesn't yet exist).
Gerrit has the facility so that we could inject the dashboard content
for this in Gerrit in a little table somewhere, but the data would
fundamentally live outside of Gerrit. It would also mean that all the
aggregate reporting of 3rd Party CI that's being done in custom gerrit
scripts, could be integrated directly into such a thing.

Agreed, it would be a great improvement in usability if we stopped all
CI systems, including our default Jenkins, from ever commenting on
reviews. At most gating CIs should +1/-1. Having a table of results
displayed, pulling the data from an external result tracking system
would be a great idea.

Even better if this external system had a nice button you can press
to trigger re-check, so we can stop using comments for that too.

To me the ideal world is where the only things adding comments to
reviews are human and their comments are actually about the code
in the patch :-)

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/
:|
|: http://libvirt.org -o- http://virt-manager.org
:|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/
:|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc
:|


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jun 27, 2014 by john.griffith_at_sol (4,960 points)   1 2 3
0 votes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

What if 3rd Party CI didn't vote in Gerrit? What if it instead
published to some 3rd party test reporting site (a thing that
doesn't yet exist). Gerrit has the facility so that we could inject
the dashboard content for this in Gerrit in a little table
somewhere, but the data would fundamentally live outside of Gerrit.
It would also mean that all the aggregate reporting of 3rd Party CI
that's being done in custom gerrit scripts, could be integrated
directly into such a thing.

If it really does show up right in Gerrit as if it were integrated,
then that would be fine with me. I think the biggest problem we have
right now is that a lot of the CI systems are very inconsistent in
their reporting and we often don't realize when one of them hasn't
voted. If the new thing could fill out the chart based on everything
we expect to see a vote from, so that it's very clear that one is
missing, then that's a net win right there.

But at the end of the day, we need to make sure we're improving
visibility and not hiding all the noise they generate just because
it's annoying. As long as something like what was suggested here is
injected right into the gerrit API, then I'm in favor of something
like this.

iQEcBAEBAgAGBQJTrar8AAoJEBeZxaMESjNVizsH/RfjoJOlX80N+jviDHG4yMUv
n9dRV2vFfitKmapN8HIWdo6aFjRDHb5t3BWcO+9xP7+w/nZYpC88Y6Eq8c8HeZPz
WIeVVY8/AmL/VVGXNojeIoCdszR8vdlaEaOrlKCjQIr8zqZJrz6LgPXIfoVCVrMg
hHN2+TM4rn9scFHlKSnf8vq7rLj/aTfzG+ITSSksslM2YxWkqDFWWBTrTnqrWHx5
Uk91JSFmWTC2UnbyVzPI88D0a7HGkpyxgFnWb8Y63RBr/JygFisoBLaqbHA9o9ML
ajI+st8ySBEwWGyVkckzX5Go1KWgTP9pH1zwhpZz+qvEZCJA6Pbp16+r+kTFBew=
=NkTB
-----END PGP SIGNATURE-----

responded Jun 27, 2014 by Dan_Smith (9,860 points)   1 2 4
0 votes

Sean, there is a proposal[1] in upstream gerrit to fix this issue. David's
proposal is to make Gerrit handle multiple notifications channels per
project which would allow us to treat bot notifications differently than
human notifications. He has the same problem as we do, most of his builds
are done using 'third party' CI so he has many more third party systems
reporting to his gerrit instance than we do. Please take a look and
provide feedback.

[1] https://code.google.com/p/gerrit/issues/detail?id=2465
https://code.google.com/p/gerrit/issues/detail?id=2465

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jun 27, 2014 by Zaro (1,280 points)   2
0 votes

On 06/27/2014 02:19 PM, Zaro wrote:

Sean, there is a proposal[1] in upstream gerrit to fix this issue.
David's proposal is to make Gerrit handle multiple notifications
channels per project which would allow us to treat bot notifications
differently than human notifications. He has the same problem as we do,
most of his builds are done using 'third party' CI so he has many more
third party systems reporting to his gerrit instance than we do. Please
take a look and provide feedback.

[1] _https://code.google.com/p/gerrit/issues/detail?id=2465_

That seems to require the new Change Screen?

-Sean

--
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL:

responded Jun 27, 2014 by Sean_Dague (66,200 points)   4 8 14
0 votes

David did suggest adding REST api endpoints to get data for each channel so
it doesn't necessarily require you to even use the gerrit web ui. However
the web UI's old screen is pretty much dead so I assume the presentation
would only be available in new change screen. I know Openstackers have had
reservations about the new change screen but please don't judge it by
trying the gerrit 2.8.x implementation. It has improved quite a bit. I
have been using it on the https://gerrit-review.googlesource.com and do
prefer it over the old change screen :)

-Khai

On Fri, Jun 27, 2014 at 6:38 PM, Sean Dague wrote:

On 06/27/2014 02:19 PM, Zaro wrote:

Sean, there is a proposal[1] in upstream gerrit to fix this issue.
David's proposal is to make Gerrit handle multiple notifications
channels per project which would allow us to treat bot notifications
differently than human notifications. He has the same problem as we do,
most of his builds are done using 'third party' CI so he has many more
third party systems reporting to his gerrit instance than we do. Please
take a look and provide feedback.

[1] _https://code.google.com/p/gerrit/issues/detail?id=2465_

That seems to require the new Change Screen?

    -Sean

--
Sean Dague
http://dague.net


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Jun 27, 2014 by Zaro (1,280 points)   2
0 votes

I do quite like the conflicts with part, I was thinking about that the
other day. That would be hugely useful.

-Sean

On 06/27/2014 02:56 PM, Zaro wrote:
David did suggest adding REST api endpoints to get data for each channel
so it doesn't necessarily require you to even use the gerrit web ui.
However the web UI's old screen is pretty much dead so I assume the
presentation would only be available in new change screen. I know
Openstackers have had reservations about the new change screen but
please don't judge it by trying the gerrit 2.8.x implementation. It has
improved quite a bit. I have been using it on
the https://gerrit-review.googlesource.com and do prefer it over the
old change screen :)

-Khai

On Fri, Jun 27, 2014 at 6:38 PM, Sean Dague > wrote:

On 06/27/2014 02:19 PM, Zaro wrote:
>
> Sean, there is a proposal[1] in upstream gerrit to fix this issue.
>  David's proposal is to make Gerrit handle multiple notifications
> channels per project which would allow us to treat bot notifications
> differently than human notifications.  He has the same problem as
we do,
> most of his builds are done using 'third party' CI so he has many more
> third party systems reporting to his gerrit instance than we do.
 Please
> take a look and provide feedback.
>
> [1] _https://code.google.com/p/gerrit/issues/detail?id=2465_

That seems to require the new Change Screen?

        -Sean

--
Sean Dague
http://dague.net


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL:

responded Jun 27, 2014 by Sean_Dague (66,200 points)   4 8 14
0 votes

Sean Dague writes:

It seems what we actually want is a dashboard of these results. We want
them available when we go to Gerrit, but we don't want them in Gerrit
itself.

I agree with most of what you wrote, particularly that we want them
available in Gerrit and with a sensible presentation. Though I don't
think that necessarily excludes storing the data in Gerrit, as Khai
points out.

I think one ideal interface is a table in Gerrit of all the jobs run on
a change (from all systems) and their results (with links). That sounds
like the project David is working on (that Khai pointed out), which
isn't surprising -- we sketched out the initial design with him about
three years ago; our needs are very similar.

David's proposal is a table like:

Job/Category | started on | ended on | href | status
foo [...]
bar [...]
bat [?]

I think that's a great approach because it provides the needed
information to reviewers in an appropriate interface in Gerrit.

I suspect that whether the data are stored in Gerrit or a different
system, the (future) vinz web ui could retrieve it from either.

An alternate approach would be to have third-party CI systems register
jobs with OpenStack's Zuul rather than using their own account. This
would mean only a single report of all jobs (upstream and 3rd-party)
per-patchset. It significantly reduces clutter and makes results more
accessible -- but even with one system we've never actually wanted to
have Jenkins results in comments, so I think one of the other options
would be preferred. Nonetheless, this is possible with a little bit of
work.

If anyone is seriously interested in working on Sean's proposal or
something similar, please write up a spec in the
openstack-infra/infra-specs repo. If you'd like to help with David's
proposal, the upstream Gerrit project is the best place to collaborate.

-Jim

responded Jun 28, 2014 by James_E._Blair (5,080 points)   1 5 7
0 votes

Sean Dague writes:

This has all gone far enough that someone actually wrote a Grease Monkey
script to purge all the 3rd Party CI content out of Jenkins UI. People
are writing mail filters to dump all the notifications. Dan Berange
filters all them out of his gerrit query tools.

I should also mention that there is a pending change to do something
similar via site-local Javascript in our Gerrit:

https://review.openstack.org/#/c/95743/

I don't think it's an ideal long-term solution, but if it works, we may
have some immediate relief without all having to install greasemonkey
scripts.

-Jim

responded Jun 28, 2014 by James_E._Blair (5,080 points)   1 5 7
...