settingsLogin | Registersettings

[openstack-dev] [stable] Making stable maintenance its own OpenStack project team

0 votes

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which was a
bit of a frankenteam of the things I happened to be leading: release
management, stable branch maintenance and vulnerability management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic was not
the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we should
spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point release
    management, but as of stable/liberty this is now done by the release
    management team and triggered by the project-specific stable maintenance
    teams, so there is no more overlap in tooling used there

  • Following the kilo reform, the stable team is now focused on defining
    and enforcing a common stable branch policy[1], rather than approving
    every patch. Being more visible and having more dedicated members can
    only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused on
    release management and does not have the history I had with stable
    branch policy. So it might be the right moment to refocus release
    management solely on release management and get the stable team its own
    leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources being
    dedicated to it

  • If the team expands, it could finally own stable branch health and
    gate fixing. If that ends up all falling under the same roof, that team
    could make decisions on support timeframes as well, since it will be the
    primary resource to make that work

So.. good idea ? bad idea ? What do current stable-maint-core[2] members
think of that ? Who thinks they could step up to lead that team ?

[1] http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 9, 2015 in openstack-dev by Thierry_Carrez (57,480 points)   3 8 13

29 Responses

0 votes

On 11/11/15 13:39 -0600, Matt Riedemann wrote:

On 11/11/2015 8:51 AM, Flavio Percoco wrote:

On 09/11/15 21:30 -0600, Matt Riedemann wrote:

On 11/9/2015 9:12 PM, Matthew Treinish wrote:

On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:

On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:

Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:

On 11/9/2015 10:41 AM, Thierry Carrez wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which
was a bit of a frankenteam of the things I happened to be leading:
release management, stable branch maintenance and vulnerability
management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic
was not the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we
should spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point
    release management, but as of stable/liberty this is now done by
    the release management team and triggered by the project-specific
    stable maintenance teams, so there is no more overlap in tooling
    used there

  • Following the kilo reform, the stable team is now focused on
    defining and enforcing a common stable branch policy[1], rather
    than approving every patch. Being more visible and having more
    dedicated members can only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused
    on release management and does not have the history I had with
    stable branch policy. So it might be the right moment to refocus
    release management solely on release management and get the stable
    team its own leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources
    being dedicated to it

  • If the team expands, it could finally own stable branch health
    and gate fixing. If that ends up all falling under the same roof,
    that team could make decisions on support timeframes as well,
    since it will be the primary resource to make that work

Isn't this kind of already what the stable maint team does? Well,
that and some QA people like mtreinish and sdague.

So.. good idea ? bad idea ? What do current stable-maint-core[2]
members think of that ? Who thinks they could step up to lead that
team ?

[1]
http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

With the decentralizing of the stable branch stuff in Liberty [1] it
seems like there would be less use for a PTL for stable branch
maintenance - the cats are now herding themselves, right? Or at
least that's the plan as far as I understood it. And the existing
stable branch wizards are more or less around for help and answering
questions.

The same might be said about releasing from master and the release
management team. There's still some benefit to having people
dedicated
to making sure projects all agree to sane policies and to keep up
with
deliverables that need to be released.

Except the distinction is that relmgt is actually producing
something. Relmgt
has the releases repo which does centralize library releases, reno
to do the
release notes, etc. What does the global stable core do? Right now
it's there
almost entirely to just add people to the project specific stable
core teams.

-Matt Treinish

I'd like to move the discussion from what are the roles of the
current stable-maint-core and more towards what the benefits would
be having a stable-maint team rather than the -core group alone.

Personally I think the stable maintenance should be quite a lot more
than unblocking gate and approving people allowed to merge to the
stable branches.

Sure, but that's not we're talking about here are we? The other
tasks, like
backporting changes for example, have been taken on by project teams.
Even in
your other email you mentioned that you've been doing backports and
other tasks
that you consider stable maint in a glance only context. That's
something we
changed in kilo which ttx referenced in [1] to enable that to happen,
and it was
the only way to scale things.

The discussion here is about the cross project effort around stable
branches,
which by design is a more limited scope now. Right now the cross
project effort
around stable branch policy is really 2 things (both of which ttx
already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

The only lever on #2 is that the global stable-maint-core is the only
group
which has add permissions to the per project stable core groups.
(also the
stable branch policy wiki, but that rarely changes) We specifically
shrunk it to
these 2 things in. [1] Well, really 3 things there, but since we're
not doing
integrated stable point releases in the future its now only 2.

This is my whole argument that creating a new team doesn't do
anything. Living
under rel-mgt, some other project, or creating a new one isn't going
to change
the fact that cross-project stable maint is the same 2 tasks which
basically
nobody cares about, which TBH your email is just an indication of.
Even if we
wanted to grow to something beyond these 2 tasks they would still be
the core of
whatever it becomes and a lack of people caring about them will just
block any
potential scope growth.

Frankly, this is the problem with any ML or open discussion about
stable branch
maint. Everyone has an opinion about everything but lacks actual
context or
never steps up to work on the cross-project side of this. I don't
think creating
a new team that has no repos or other artifacts and therefore no
stackalytics
credit (and by extension corporate chest thumping) is magically going
to create
people actually working on this.

The logical follow on idea to the above is to create a stable branch
policy doc
repo and a project team to own that. But, I still don't think that
solves the
problem, especially since the real priority issue in this space is #1
which
at most is a handful of us bothering to look at that, (well that and
the stable
policy barely ever changes) which honestly really isn't the majority
of the
stable-maint-core group. We also have a similar problem with gate
issues on
master and it's mostly the same people looking at these things there
too.

IMHO the only way for creating a separate project team to be useful
is to
re-centralize more responsibilities of stable maint, which is
something I don't
think we should do because we'll just hit the same scaling issues we
had before
kilo which prompted that change in the first place. It's the same
problem every
horizontal, cross project, or whatever you want to call it effort has
with
OpenStack's growth and why most have moved to the distributed self
service
models.

But, if the goal is just to not make Doug responsible for #2, which
is something
ttx was primarily doing before, then I guess it's fine but we should
be honest
about it and just make ttx the new team leader. :) I honestly don't
think
anything else discussed will change by explicitly making it a
separate team.

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078
281.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Matty T got all P&V in Tokyo! :)

I pretty much agree with everything said above. I don't think we make
many policy changes, the stable maint core team is already pretty
decentralized as it is. We rarely seem to communicate much except when
the (1) gate is wedged or (2) we're coming up on a release and need a
push for reviews.

And this is perhaps one of the problems. I don't think decentralizing
the stable team is enough to have proper stable maintenance. It's a
great step forward and I don't think the proposal is to remove that.

Just like for the releases (especially for projects following
coordinated releases), stable branches could benefit from having
someone looking after them (for those projects that apply to the tag
ttx mentioned in one of his previous emails) and coordinate releases,
status, etc.

I think there's more to it than enforcing policies and keeping the
gate sane. What about projects w/o backports? Are we encouraging ppl
to do enough backports to stable branches?

The small group of gate thumpers raise hell when (1) comes up, and the
stable branch champion coordinates (2).

I've asked this in other threads before, but do we need a PTL for
this? I'm assuming ttx doesn't want this since he's asking for
volunteers. Doug doesn't want it because of release mgmt duties. But
what does a stable branch PTL do? If we have policy changes on stable
that need discussing, can't we come together and discuss those as
needed as a group?

Here's a couple of things that I'd love to see the volunteer for that
job to do:

  • Help coordinating stable releases when they are needed.

But isn't this why we decentralized the stable release stuff? So you
don't need a single person or small team herding all the cats?

Perhaps, but still some coordination is needed. I believe.

  • Help creating tools to monitor existing backports.

I'm not sure a PTL is needed for this. And I don't think a tool is
needed, we have gerrit. With decentralized stable maint to the
projects (including releasing), if I'm a nova person that cares about
stable (and I am), then I can find out what's open for stable/liberty
backports using gerrit:

I meant something that would also help enforcing the policies and what
not. I know what you can do w/ gerrit.

  • Help creating tools that could help identifying patches that might
    be worth backporting.

Again, I'm not sure a PTL is needed for this, just some hackers. I
could see value in writing a script that scans launchpad for things
like 'liberty-backport-potential' where the fix is merged in mitaka
but not proposed to stable/liberty.

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty,n,z

Easy peasy. We actually have these same dashboards linked into our
weekly nova meeting agenda just for awareness.

Again, I know what you can do with gerrit. What I was suggesting is a
script that would help scan launchpads as you mentioned.

I'm not saying you need a PTL to work on that but, we've seen
(especially with the growth of the number of liaisons) that someone
that can act as a representative is, most of the time, helpful. :)

  • Be a contact for whenever things break in stable branches and
    coordinate with folks that can help fixing those issues.

We have the #openstack-stable channel, people can ask for help there.
We have the mailing list where stable blocking issues are regularly
brought forward to the wider audience. I guess being PTL would mean
there is a lightning rod when something goes bad, but I'd think the
majority of the time they are going to defer to the PTL of whatever
project is blocked and get them to work on it, i.e. if trove
stable/liberty is blocked b/c of some issue in their own CI tooling.

Again, it's not a matter of support but coordination, tooling and
enforcement.

Flavio

Not sure if all the above points are worth it but I do see this as an
useful role to have.

Cheers,
Flavio

And for gate issues, the people that generally work on those are doing
it because it benefits them, either because it keeps them from having
to maintain forks out of tree or because it gets other changes landed
that are needed for their product/customers. I don't see a new team
making a difference in attracting new people to really dig into the
ugly job of unwedging the gate on stable - it takes quite a bit of
time to get up to speed. Tony is a great example of someone that has
kind of come out of left field and said he wants to help with this and
has stuck with it long enough that he makes a difference and it's
worth spending time mentoring him on the stable gate issues, but those
people are far and few between.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 12, 2015 by Flavio_Percoco (36,960 points)   3 8 11
0 votes

So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend regular
meetings about it and to lurk on #openstack-stable. I'm fine helping the
team in the spin-off effort but I don't want to lead it (I proved I was
unable to make it my top priority in the past, so I think the team
deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

Once we have a list of key members we should set up a meeting to discuss
the details.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 13, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

Excerpts from Thierry Carrez's message of 2015-11-13 15:10:24 +0100:

So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend regular
meetings about it and to lurk on #openstack-stable. I'm fine helping the
team in the spin-off effort but I don't want to lead it (I proved I was
unable to make it my top priority in the past, so I think the team
deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

Once we have a list of key members we should set up a meeting to discuss
the details.

I'm willing to help with the initial team setup, working out the
details, transitioning ownership of docs, etc. I'm not sure how much
time I'll have to devote to other stable-related activities this cycle,
but may have more in the future.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 13, 2015 by Doug_Hellmann (87,520 points)   3 4 12
0 votes

On 13/11/15 15:10 +0100, Thierry Carrez wrote:

So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend regular
meetings about it and to lurk on #openstack-stable. I'm fine helping the
team in the spin-off effort but I don't want to lead it (I proved I was
unable to make it my top priority in the past, so I think the team
deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

I'm happy to join with obvious limited time. I've already worked w/
Erno on some of this stuff.

As long as Ernoi (or someone else) is still willing to lead this
effort, I think we can give this a try.

Flavio

Once we have a list of key members we should set up a meeting to discuss
the details.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 16, 2015 by Flavio_Percoco (36,960 points)   3 8 11
0 votes

On 13 November 2015 at 14:10, Thierry Carrez thierry@openstack.org wrote:
So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend regular
meetings about it and to lurk on #openstack-stable. I'm fine helping the
team in the spin-off effort but I don't want to lead it (I proved I was
unable to make it my top priority in the past, so I think the team
deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

Once we have a list of key members we should set up a meeting to discuss
the details.

I think this summary is pretty reflective of the thread so far. I
also agree that I see very little benefit from transitioning the
stable-maint-core team into a recognised project team, I can't imagine
there being any measurable benefit from this.. feels more like a paper
exercise.. however, I don't feel strongly enough to try and stop
this.

Due to limited time, I'd not like to be a driver... but would like to
be part of the seed group of cores. If this is to happen, it seems
reasonable to map stable-maint-core -> stable-maint project.

We've traditionally been pretty bad with discussion and engaging with
fellow stable-maint members, so a standing (short) meeting might help
improve this.

--
Kind Regards,
Dave Walker


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 16, 2015 by Dave_Walker (3,660 points)   1 4
0 votes

On 11/13/2015 8:10 AM, Thierry Carrez wrote:
So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend regular
meetings about it and to lurk on #openstack-stable. I'm fine helping the
team in the spin-off effort but I don't want to lead it (I proved I was
unable to make it my top priority in the past, so I think the team
deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

Once we have a list of key members we should set up a meeting to discuss
the details.

I've been doing some thinking in the last week and I'm going to throw my
hat in the ring for leading this. I've definitely played devil's
advocate in this thread, but I think it also points out several issues
that are going to require some focused work and communication and I
think I'm suited to lead that.

Background:

  • I've been stable-maint-core since April. I primarily focus on nova
    stable branches but also get into other projects when necessary for
    critical bugs and blocking gate issues.
  • Much of my focus is on the CI issues, i.e. Tony's tangled web of
    onions. I already deal enough in QA on trunk that it's a natural
    extension to stable branches.
  • python-novaclient release liaison.
  • I already do quite a bit of stable branch maintenance internally
    (which is how I got involved with it upstream too). So I have incentive
    to keep it working.

This is basically what I see for PTL of a stable maint team:

  1. Helping projects deal with stable branch policy and process. We are
    decentralizing now but I suspect there will still be a lot of learning
    and cat herding needed in individual projects. This includes being
    around to help (I'm always in #openstack-stable) and helping to
    coordinate point releases as needed.

  2. Keeping the CI system working on stable. This is already something
    I've been doing on stable for a long time so that just continues. It
    also means keeping a finger on how the periodic nightly jobs are doing
    and getting projects involved/aware when their stable branches are
    failing (bringing that up in the mailing list, project meetings or a
    cross project meeting as needed). An extension of this is seeing what we
    can do to try and keep stable branches around longer, which is clearly
    something the operator community has been wanting.

  3. Mentoring. A big part of this thread is talking about a lack of
    resources to work on stable branch issues (usually CI). So part of
    mentoring here is identifying and helping people that are wanting to
    work in this space. A perfect example is tonyb and the work he's done
    the past several months on tracking gate failures and blocking issues.
    It also includes keeping track of who's doing a good job with reviews
    and seeing if they are ready for core.

  4. Identifying official tags for a project, which doesn't just mean they
    have a stable branch but that they abide by the stable branch policy
    (what's appropriate to backport) and that they are maintaining their
    stable branches; this means actually reviewing proposed changes, being
    proactive about backporting high severity fixes, and keeping their CI
    jobs clean.

  5. Looking for ways to improve processes and tooling. One thing that has
    come up in this thread was a need for tooling to identify backportable
    changes (e.g. a high severity bug is marked as backport potential in
    launchpad but no one has done the backport yet). I'm also not entirely
    sure we have something today that is tracking stable branch review stats
    so that we can identify people that are doing (or cores not doing) reviews.

So this is a bit long-winded but it's what I see as being important for
a separate team and PTL to do in this space. If we're going to do this,
I want to make sure these things are covered and I think I'm in a
position to do that.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 17, 2015 by Matt_Riedemann (48,320 points)   3 9 22
0 votes

Matt Riedemann wrote:
[...]
This is basically what I see for PTL of a stable maint team:

  1. Helping projects deal with stable branch policy and process. We are
    decentralizing now but I suspect there will still be a lot of learning
    and cat herding needed in individual projects. This includes being
    around to help (I'm always in #openstack-stable) and helping to
    coordinate point releases as needed.

  2. Keeping the CI system working on stable. This is already something
    I've been doing on stable for a long time so that just continues. It
    also means keeping a finger on how the periodic nightly jobs are doing
    and getting projects involved/aware when their stable branches are
    failing (bringing that up in the mailing list, project meetings or a
    cross project meeting as needed). An extension of this is seeing what we
    can do to try and keep stable branches around longer, which is clearly
    something the operator community has been wanting.

  3. Mentoring. A big part of this thread is talking about a lack of
    resources to work on stable branch issues (usually CI). So part of
    mentoring here is identifying and helping people that are wanting to
    work in this space. A perfect example is tonyb and the work he's done
    the past several months on tracking gate failures and blocking issues.
    It also includes keeping track of who's doing a good job with reviews
    and seeing if they are ready for core.

  4. Identifying official tags for a project, which doesn't just mean they
    have a stable branch but that they abide by the stable branch policy
    (what's appropriate to backport) and that they are maintaining their
    stable branches; this means actually reviewing proposed changes, being
    proactive about backporting high severity fixes, and keeping their CI
    jobs clean.

  5. Looking for ways to improve processes and tooling. One thing that has
    come up in this thread was a need for tooling to identify backportable
    changes (e.g. a high severity bug is marked as backport potential in
    launchpad but no one has done the backport yet). I'm also not entirely
    sure we have something today that is tracking stable branch review stats
    so that we can identify people that are doing (or cores not doing) reviews.

So this is a bit long-winded but it's what I see as being important for
a separate team and PTL to do in this space. If we're going to do this,
I want to make sure these things are covered and I think I'm in a
position to do that.

This is a very good summary. I think it's critical to have good
representation of the "gate fixing" part of the role: without it, the
stable team is basically useless and totally dependent on another group,
with all the tension we had in the past. Growing the next generation of
gate fixers is essential.

I'm mostly away this week at a conference, but now that we have a list
of names, we should set up a meeting early next week to further discuss
team goals and initial leadership.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 18, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

Thierry Carrez wrote:
[...]
I'm mostly away this week at a conference, but now that we have a list
of names, we should set up a meeting early next week to further discuss
team goals and initial leadership.

We'll be holding a preliminary meeting today (Monday) at 15:00 UTC in

openstack-meeting-4, for those interested.

Regards,

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 23, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

I had few related experience working with Erno for releasing/stable branching and I have to say he significantly clarified/fixed
many issue we had.

So ++

-----Original Message-----
From: Flavio Percoco [mailto:flavio@redhat.com]
Sent: 16 November 2015 17:29
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [stable] Making stable maintenance its own OpenStack project team

On 13/11/15 15:10 +0100, Thierry Carrez wrote:

So.. quick summary of this thread so far.

(-)
* Not enough work to warrant a designated "team", now that the work is
decentralized and the cats are mostly herding themselves
* The change is unlikely to bring a meaningful improvement to the
situation, or sudden new resources

(+)
* An empowered team could tackle new coordination tasks, like engaging
more directly in converging stable branch rules across teams, or
producing tools
* Release management doesn't overlap anymore with stable branch, so
having them under that PTL is limiting and inefficient
* Reinforcing the branding (by giving it its own team) may encourage
more organizations to affect new resources to it

In summary, I think this is worth a try. If the team fails, at least it
will be on its own rather than as the 5th squeaky wheel of release
management (where lack of leadership and focus could be rightly blamed
for failure).

For this to succeed, we need someone to own the effort and push it
forward, and a number of people caring enough about it to attend
regular meetings about it and to lurk on #openstack-stable. I'm fine
helping the team in the spin-off effort but I don't want to lead it (I
proved I was unable to make it my top priority in the past, so I think
the team deserves a more focused lead).

Erno has volunteered to lead. Ihar, Alan (Pevec) and myself don't have
enough time to lead but are happy to help. Anyone else interested to
join that initial group ? Flavio ? Matt ?

I'm happy to join with obvious limited time. I've already worked w/ Erno on some of this stuff.

As long as Ernoi (or someone else) is still willing to lead this effort, I think we can give this a try.

Flavio

Once we have a list of key members we should set up a meeting to
discuss the details.

--
Thierry Carrez (ttx)


___ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 23, 2015 by fausto.marzi_at_hpe. (300 points)  
...