settingsLogin | Registersettings

[openstack-dev] [stable] Making stable maintenance its own OpenStack project team

0 votes

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which was a
bit of a frankenteam of the things I happened to be leading: release
management, stable branch maintenance and vulnerability management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic was not
the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we should
spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point release
    management, but as of stable/liberty this is now done by the release
    management team and triggered by the project-specific stable maintenance
    teams, so there is no more overlap in tooling used there

  • Following the kilo reform, the stable team is now focused on defining
    and enforcing a common stable branch policy[1], rather than approving
    every patch. Being more visible and having more dedicated members can
    only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused on
    release management and does not have the history I had with stable
    branch policy. So it might be the right moment to refocus release
    management solely on release management and get the stable team its own
    leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources being
    dedicated to it

  • If the team expands, it could finally own stable branch health and
    gate fixing. If that ends up all falling under the same roof, that team
    could make decisions on support timeframes as well, since it will be the
    primary resource to make that work

So.. good idea ? bad idea ? What do current stable-maint-core[2] members
think of that ? Who thinks they could step up to lead that team ?

[1] http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 9, 2015 in openstack-dev by Thierry_Carrez (57,480 points)   3 8 13

29 Responses

0 votes

On 11/9/2015 9:12 PM, Matthew Treinish wrote:
On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:

On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:

Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:

On 11/9/2015 10:41 AM, Thierry Carrez wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which
was a bit of a frankenteam of the things I happened to be leading:
release management, stable branch maintenance and vulnerability
management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic
was not the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we
should spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point
    release management, but as of stable/liberty this is now done by
    the release management team and triggered by the project-specific
    stable maintenance teams, so there is no more overlap in tooling
    used there

  • Following the kilo reform, the stable team is now focused on
    defining and enforcing a common stable branch policy[1], rather
    than approving every patch. Being more visible and having more
    dedicated members can only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused
    on release management and does not have the history I had with
    stable branch policy. So it might be the right moment to refocus
    release management solely on release management and get the stable
    team its own leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources
    being dedicated to it

  • If the team expands, it could finally own stable branch health
    and gate fixing. If that ends up all falling under the same roof,
    that team could make decisions on support timeframes as well,
    since it will be the primary resource to make that work

Isn't this kind of already what the stable maint team does? Well,
that and some QA people like mtreinish and sdague.

So.. good idea ? bad idea ? What do current stable-maint-core[2]
members think of that ? Who thinks they could step up to lead that
team ?

[1]
http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

With the decentralizing of the stable branch stuff in Liberty [1] it
seems like there would be less use for a PTL for stable branch
maintenance - the cats are now herding themselves, right? Or at
least that's the plan as far as I understood it. And the existing
stable branch wizards are more or less around for help and answering
questions.

The same might be said about releasing from master and the release
management team. There's still some benefit to having people dedicated
to making sure projects all agree to sane policies and to keep up with
deliverables that need to be released.

Except the distinction is that relmgt is actually producing something. Relmgt
has the releases repo which does centralize library releases, reno to do the
release notes, etc. What does the global stable core do? Right now it's there
almost entirely to just add people to the project specific stable core teams.

-Matt Treinish

I'd like to move the discussion from what are the roles of the current stable-maint-core and more towards what the benefits would be having a stable-maint team rather than the -core group alone.

Personally I think the stable maintenance should be quite a lot more than unblocking gate and approving people allowed to merge to the stable branches.

Sure, but that's not we're talking about here are we? The other tasks, like
backporting changes for example, have been taken on by project teams. Even in
your other email you mentioned that you've been doing backports and other tasks
that you consider stable maint in a glance only context. That's something we
changed in kilo which ttx referenced in [1] to enable that to happen, and it was
the only way to scale things.

The discussion here is about the cross project effort around stable branches,
which by design is a more limited scope now. Right now the cross project effort
around stable branch policy is really 2 things (both of which ttx already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

The only lever on #2 is that the global stable-maint-core is the only group
which has add permissions to the per project stable core groups. (also the
stable branch policy wiki, but that rarely changes) We specifically shrunk it to
these 2 things in. [1] Well, really 3 things there, but since we're not doing
integrated stable point releases in the future its now only 2.

This is my whole argument that creating a new team doesn't do anything. Living
under rel-mgt, some other project, or creating a new one isn't going to change
the fact that cross-project stable maint is the same 2 tasks which basically
nobody cares about, which TBH your email is just an indication of. Even if we
wanted to grow to something beyond these 2 tasks they would still be the core of
whatever it becomes and a lack of people caring about them will just block any
potential scope growth.

Frankly, this is the problem with any ML or open discussion about stable branch
maint. Everyone has an opinion about everything but lacks actual context or
never steps up to work on the cross-project side of this. I don't think creating
a new team that has no repos or other artifacts and therefore no stackalytics
credit (and by extension corporate chest thumping) is magically going to create
people actually working on this.

The logical follow on idea to the above is to create a stable branch policy doc
repo and a project team to own that. But, I still don't think that solves the
problem, especially since the real priority issue in this space is #1 which
at most is a handful of us bothering to look at that, (well that and the stable
policy barely ever changes) which honestly really isn't the majority of the
stable-maint-core group. We also have a similar problem with gate issues on
master and it's mostly the same people looking at these things there too.

IMHO the only way for creating a separate project team to be useful is to
re-centralize more responsibilities of stable maint, which is something I don't
think we should do because we'll just hit the same scaling issues we had before
kilo which prompted that change in the first place. It's the same problem every
horizontal, cross project, or whatever you want to call it effort has with
OpenStack's growth and why most have moved to the distributed self service
models.

But, if the goal is just to not make Doug responsible for #2, which is something
ttx was primarily doing before, then I guess it's fine but we should be honest
about it and just make ttx the new team leader. :) I honestly don't think
anything else discussed will change by explicitly making it a separate team.

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078
281.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Matty T got all P&V in Tokyo! :)

I pretty much agree with everything said above. I don't think we make
many policy changes, the stable maint core team is already pretty
decentralized as it is. We rarely seem to communicate much except when
the (1) gate is wedged or (2) we're coming up on a release and need a
push for reviews.

The small group of gate thumpers raise hell when (1) comes up, and the
stable branch champion coordinates (2).

I've asked this in other threads before, but do we need a PTL for this?
I'm assuming ttx doesn't want this since he's asking for volunteers.
Doug doesn't want it because of release mgmt duties. But what does a
stable branch PTL do? If we have policy changes on stable that need
discussing, can't we come together and discuss those as needed as a group?

And for gate issues, the people that generally work on those are doing
it because it benefits them, either because it keeps them from having to
maintain forks out of tree or because it gets other changes landed that
are needed for their product/customers. I don't see a new team making a
difference in attracting new people to really dig into the ugly job of
unwedging the gate on stable - it takes quite a bit of time to get up to
speed. Tony is a great example of someone that has kind of come out of
left field and said he wants to help with this and has stuck with it
long enough that he makes a difference and it's worth spending time
mentoring him on the stable gate issues, but those people are far and
few between.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by Matt_Riedemann (48,320 points)   3 9 22
0 votes

Matthew Treinish wrote:
[...]
The discussion here is about the cross project effort around stable branches,
which by design is a more limited scope now. Right now the cross project effort
around stable branch policy is really 2 things (both of which ttx already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

Right. My argument is that we need to expand the work on point (2), and
then expand the work on point (1), and that empowering them by giving
them their own team can only help.

For point (2), currently the stable branch policy is not really enforced
as much as it should. I think it's important to have a standard
definition across OpenStack of what a "stable branch" means in terms of
acceptable backports, support periods, frequency of point releases, etc.
With the big tent we see a lot of variance in the stable branch policy,
and what used to be mostly self-policed is likely to now require a lot
more attention from policy overlords.

I'm not seeing that happening with the current team structure, which is
why I proposed the change. Once empowered with their own team, it will
be easier to justify dedicating resources to it, and I hope that will go
all the way up to owning gate fixing when stable branch breaks (point
(1) which is currently mostly done off-team).

The only lever on #2 is that the global stable-maint-core is the only group
which has add permissions to the per project stable core groups. (also the
stable branch policy wiki, but that rarely changes)

Actually we'll likely create a "follows-stable-policy" tag that would be
granted by this team. I expect that to be a good lever as well.

[...]
This is my whole argument that creating a new team doesn't do anything. Living
under rel-mgt, some other project, or creating a new one isn't going to change
the fact that cross-project stable maint is the same 2 tasks which basically
nobody cares about, which TBH your email is just an indication of. Even if we
wanted to grow to something beyond these 2 tasks they would still be the core of
whatever it becomes and a lack of people caring about them will just block any
potential scope growth.

I have evidence that making that group its own project team will result
in additional resources, and increase of focus from existing resources.
The alternative is to kill stable branches altogether. But as the other
thread shows, there is still interest in them, so providing the right
vessel for resources to be dedicated to it is still worth a try.

I'm arguing that one of the reasons "nobody cares" is because it's
always been the 5th wheel of release management, and making it a
first-class citizen could unlock the situation. I don't see harm in
trying. Do you ?

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 10, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

-----Original Message-----
From: Matthew Treinish [mailto:mtreinish@kortar.org]
Sent: Tuesday, November 10, 2015 3:12 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [stable] Making stable maintenance its own
OpenStack project team

On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:

On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:

Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:

On 11/9/2015 10:41 AM, Thierry Carrez wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team
which was a bit of a frankenteam of the things I happened to be
leading:
release management, stable branch maintenance and
vulnerability
management.
While you could argue that there was some overlap between
those functions (as in, "all these things need to be
released") logic was not the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of
the Release Cycle Management team and joined there. Now I
think we should spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point
    release management, but as of stable/liberty this is now done
    by the release management team and triggered by the
    project-specific stable maintenance teams, so there is no more
    overlap in tooling used there

  • Following the kilo reform, the stable team is now focused on
    defining and enforcing a common stable branch policy[1],
    rather than approving every patch. Being more visible and
    having more dedicated members can only help in that very
    specific mission

  • The release team is now headed by Doug Hellmann, who is
    focused on release management and does not have the history I
    had with stable branch policy. So it might be the right moment
    to refocus release management solely on release management and
    get the stable team its own leadership

  • Empowering that team to make its own decisions, giving it
    more visibility and recognition will hopefully lead to more
    resources being dedicated to it

  • If the team expands, it could finally own stable branch
    health and gate fixing. If that ends up all falling under the
    same roof, that team could make decisions on support
    timeframes as well, since it will be the primary resource to
    make that work

Isn't this kind of already what the stable maint team does?
Well, that and some QA people like mtreinish and sdague.

So.. good idea ? bad idea ? What do current
stable-maint-core[2] members think of that ? Who thinks they
could step up to lead that
team ?

[1]
http://docs.openstack.org/project-team-guide/stable-branches.h
tml [2]
https://review.openstack.org/#/admin/groups/530,members

With the decentralizing of the stable branch stuff in Liberty
[1] it seems like there would be less use for a PTL for stable
branch maintenance - the cats are now herding themselves, right?
Or at least that's the plan as far as I understood it. And the
existing stable branch wizards are more or less around for help
and answering
questions.

The same might be said about releasing from master and the release
management team. There's still some benefit to having people
dedicated to making sure projects all agree to sane policies and
to keep up with deliverables that need to be released.

Except the distinction is that relmgt is actually producing
something. Relmgt has the releases repo which does centralize
library releases, reno to do the release notes, etc. What does the
global stable core do? Right now it's there almost entirely to just add
people to the project specific stable core teams.

-Matt Treinish

I'd like to move the discussion from what are the roles of the current
stable-maint-core and more towards what the benefits would be having a
stable-maint team rather than the -core group alone.

Personally I think the stable maintenance should be quite a lot more than
unblocking gate and approving people allowed to merge to the stable
branches.

Sure, but that's not we're talking about here are we? The other tasks, like
backporting changes for example, have been taken on by project teams.
Even in your other email you mentioned that you've been doing backports
and other tasks that you consider stable maint in a glance only context. That's
something we changed in kilo which ttx referenced in [1] to enable that to
happen, and it was the only way to scale things.

The discussion here is about the cross project effort around stable branches,
which by design is a more limited scope now. Right now the cross project
effort around stable branch policy is really 2 things (both of which ttx already
mentioned):

  1. Keeping the gates working on the stable branches 2. Defining and
    enforcing stable branch policy.

The only lever on #2 is that the global stable-maint-core is the only group
which has add permissions to the per project stable core groups. (also the
stable branch policy wiki, but that rarely changes) We specifically shrunk it to
these 2 things in. [1] Well, really 3 things there, but since we're not doing
integrated stable point releases in the future its now only 2.

This is my whole argument that creating a new team doesn't do anything.
Living under rel-mgt, some other project, or creating a new one isn't going to
change the fact that cross-project stable maint is the same 2 tasks which
basically nobody cares about, which TBH your email is just an indication of.
Even if we wanted to grow to something beyond these 2 tasks they would
still be the core of whatever it becomes and a lack of people caring about
them will just block any potential scope growth.

Frankly, this is the problem with any ML or open discussion about stable
branch maint. Everyone has an opinion about everything but lacks actual
context or never steps up to work on the cross-project side of this. I don't
think creating a new team that has no repos or other artifacts and therefore
no stackalytics credit (and by extension corporate chest thumping) is
magically going to create people actually working on this.

The logical follow on idea to the above is to create a stable branch policy doc
repo and a project team to own that. But, I still don't think that solves the
problem, especially since the real priority issue in this space is #1 which at
most is a handful of us bothering to look at that, (well that and the stable
policy barely ever changes) which honestly really isn't the majority of the
stable-maint-core group. We also have a similar problem with gate issues on
master and it's mostly the same people looking at these things there too.

IMHO the only way for creating a separate project team to be useful is to re-
centralize more responsibilities of stable maint, which is something I don't
think we should do because we'll just hit the same scaling issues we had
before kilo which prompted that change in the first place. It's the same
problem every horizontal, cross project, or whatever you want to call it effort
has with OpenStack's growth and why most have moved to the distributed
self service models.

But, if the goal is just to not make Doug responsible for #2, which is
something ttx was primarily doing before, then I guess it's fine but we should
be honest about it and just make ttx the new team leader. :) I honestly don't
think anything else discussed will change by explicitly making it a separate
team.

-Matt Treinish

Why not? I mean if there is interest to get stable separated from release management and having it's own team, why shouldn't we make it more effective? While I do agree that what you described is the current situation, I'd like to disagree it being the best way forward. If Thierry is willing to take the team and develop the stable efforts forward, that would be great.

What I was proposing in my first e-mail to the thread was to include the stable contributors (not just gatekeepers) to the team. Having those dedicated folks in the projects working on their project stable branches which would include them directly to the stable-maint team. Having the stable-maint-core being the core part of team still owning those -stable-maint (core) groups. What I'm looking here is common grounds around stable maintenance combining the people who cares about the stable releases under same umbrella. Hopefully more working together, sharing processes and experiences about what works and what doesn't.

In a sense that would bring [1] back centralized, but instead of trying to find the people to do that, lets just include everyone who is doing that. What comes to the statistics, they obviously can be massaged to what direction ever. Still I wouldn't mind seeing the stable efforts being listed (if that's reason for organization X to put some effort to triage bugs against stable branches and do the backporting & reviews, gr8). Personally I just fail to see how this opportunity to improve things is excluded by the current situation.

The Stable Branch Policy, lets not put it in any repo. Doing that would give indication that we expect changes into it and I'd prefer that policy mostly staying stable as well.

I guess what I'm chasing here is that I'm not the only one frustrated around totally different experience about stable releases across the projects and their state. I'd be more than happy having some framework through which I can put my efforts in coming months trying to improve that. If that means that we declare it doomed after couple of cycles, so be it, but at least I'd be willing to take the risk hoping that something good comes out of it.

  • Erno


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by kuvaja_at_hpe.com (1,320 points)   1
0 votes

Kuvaja, Erno wrote:
[...]
The Stable Branch Policy, lets not put it in any repo. Doing that would give indication that we expect changes into it and I'd prefer that policy mostly staying stable as well.

It actually already is in a repo:

http://git.openstack.org/cgit/openstack/project-team-guide/tree/doc/source/stable-branches.rst

:)

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

The Stable Branch Policy, lets not put it in any repo.
Doing that would give indication that we expect changes into it and I'd prefer that policy mostly staying stable as well.

What indication did it give when it was in wiki? :)

https://wiki.openstack.org/wiki/StableBranch should be replaced with
the link to it to avoid confusion.
There were changes since it was moved to the repo:
https://wiki.openstack.org/w/index.php?title=StableBranch&action=history
Which team approves the changes in the project-team-guide, can it be per file?

Regarding stable team spin-off, I think it's worth discussing further
to define exact scope, then see show of hands before formally setting
the team up as I'm afraid we don't have the critical mass yet.
I'm personally tied up with other obligations next few months so
cannot actively help or lead this initial effort.

Cheers,
Alan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by Alan_Pevec (5,300 points)   1 3 4
0 votes

On 11/10/2015 5:41 AM, Alan Pevec wrote:

The Stable Branch Policy, lets not put it in any repo.
Doing that would give indication that we expect changes into it and I'd prefer that policy mostly staying stable as well.

What indication did it give when it was in wiki? :)

https://wiki.openstack.org/wiki/StableBranch should be replaced with
the link to it to avoid confusion.

Done.

There were changes since it was moved to the repo:
https://wiki.openstack.org/w/index.php?title=StableBranch&action=history
Which team approves the changes in the project-team-guide, can it be per file?

Regarding stable team spin-off, I think it's worth discussing further
to define exact scope, then see show of hands before formally setting
the team up as I'm afraid we don't have the critical mass yet.
I'm personally tied up with other obligations next few months so
cannot actively help or lead this initial effort.

Cheers,
Alan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by Matt_Riedemann (48,320 points)   3 9 22
0 votes

Thierry Carrez thierry@openstack.org wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which was a
bit of a frankenteam of the things I happened to be leading: release
management, stable branch maintenance and vulnerability management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic was not
the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we should
spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point release
    management, but as of stable/liberty this is now done by the release
    management team and triggered by the project-specific stable maintenance
    teams, so there is no more overlap in tooling used there

  • Following the kilo reform, the stable team is now focused on defining
    and enforcing a common stable branch policy[1], rather than approving
    every patch. Being more visible and having more dedicated members can
    only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused on
    release management and does not have the history I had with stable
    branch policy. So it might be the right moment to refocus release
    management solely on release management and get the stable team its own
    leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources being
    dedicated to it

  • If the team expands, it could finally own stable branch health and
    gate fixing. If that ends up all falling under the same roof, that team
    could make decisions on support timeframes as well, since it will be the
    primary resource to make that work

So.. good idea ? bad idea ? What do current stable-maint-core[2] members
think of that ? Who thinks they could step up to lead that team ?

[1] http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

--
Thierry Carrez (ttx)

Lots of great thoughts in the thread, trying to catch up with it.

I lean towards supporting the split; if anything, to make responsibilities
more clear. Currently, [especially since we disintegrated the monolithic
stable-maint team into per-project teams] I struggle to reach the whole
stable team and get some final decisions on unclear matters set. F.e. I
sent several [stable] emails to this ML recently that would require some
more input from outside neutron-stable-maint that I am part of, and I don’t
see enough responses, neither a way to understand where the discussion
leaned and what my next steps are.

I believe if we have a dedicated team for the effort, we would at least be
able to get more attention to such discussions, and hopefully would be able
to make final calls on policies common to the whole project.

I am not sure we actually require PTL for that to happen, though I also
don’t see it a bad thing if we do have the role. [And if you wonder, I
won’t be able to run for PTL for the project but I am happy to help the
cause.]

I think the main responsibility of the team would be enabling other,
per-project teams to do their job. Meaning, handling common policy
questions; providing tools where needed; spearheading and monitoring
initiatives that would make stable branches less broken; providing guidance
to project teams on how to manage stable branches properly, thru docs and
irc/email support; making calls on support phases and their length; etc.

Obviously it is not a full blown software project; separating concerns
seems justified here nevertheless.

Ihar


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 10, 2015 by Ihar_Hrachyshka (35,300 points)   3 10 16
0 votes

On Tue, Nov 10, 2015 at 10:13:09AM +0100, Thierry Carrez wrote:
Matthew Treinish wrote:

[...]
The discussion here is about the cross project effort around stable branches,
which by design is a more limited scope now. Right now the cross project effort
around stable branch policy is really 2 things (both of which ttx already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

Right. My argument is that we need to expand the work on point (2), and
then expand the work on point (1), and that empowering them by giving
them their own team can only help.

Well you know my position on this well, but without people working on #1 policy
changes don't actually do a whole lot. You can look at our "experiment" in a
longer support window last year for proof of that. I don't think coming about
it from the other direction will actually change who looks at gate issues and
how we fail to scale there.

For point (2), currently the stable branch policy is not really enforced
as much as it should. I think it's important to have a standard
definition across OpenStack of what a "stable branch" means in terms of
acceptable backports, support periods, frequency of point releases, etc.
With the big tent we see a lot of variance in the stable branch policy,
and what used to be mostly self-policed is likely to now require a lot
more attention from policy overlords.

I'm not seeing that happening with the current team structure, which is
why I proposed the change. Once empowered with their own team, it will
be easier to justify dedicating resources to it, and I hope that will go
all the way up to owning gate fixing when stable branch breaks (point
(1) which is currently mostly done off-team).

The only lever on #2 is that the global stable-maint-core is the only group
which has add permissions to the per project stable core groups. (also the
stable branch policy wiki, but that rarely changes)

Actually we'll likely create a "follows-stable-policy" tag that would be
granted by this team. I expect that to be a good lever as well.

[...]
This is my whole argument that creating a new team doesn't do anything. Living
under rel-mgt, some other project, or creating a new one isn't going to change
the fact that cross-project stable maint is the same 2 tasks which basically
nobody cares about, which TBH your email is just an indication of. Even if we
wanted to grow to something beyond these 2 tasks they would still be the core of
whatever it becomes and a lack of people caring about them will just block any
potential scope growth.

I have evidence that making that group its own project team will result
in additional resources, and increase of focus from existing resources.
The alternative is to kill stable branches altogether. But as the other
thread shows, there is still interest in them, so providing the right
vessel for resources to be dedicated to it is still worth a try.

I'm arguing that one of the reasons "nobody cares" is because it's
always been the 5th wheel of release management, and making it a
first-class citizen could unlock the situation. I don't see harm in
trying. Do you ?

No, there is no harm in trying. Maybe I'm just more jaded and bitter, but I just
don't see how making it a separate thing is going to change anything. I don't
expect all the stable unicorns and fairies to come out of the forest and solve
all our problems once they finally have a separate project team. The work has
always been there a dedicated project team doesn't magically create interest
from my POV. If people were truly interested in helping they'd be contributing
already.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 10, 2015 by Matthew_Treinish (11,200 points)   2 5 5
0 votes

On 09/11/15 21:30 -0600, Matt Riedemann wrote:

On 11/9/2015 9:12 PM, Matthew Treinish wrote:

On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:

On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:

Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:

On 11/9/2015 10:41 AM, Thierry Carrez wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which
was a bit of a frankenteam of the things I happened to be leading:
release management, stable branch maintenance and vulnerability
management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic
was not the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we
should spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point
    release management, but as of stable/liberty this is now done by
    the release management team and triggered by the project-specific
    stable maintenance teams, so there is no more overlap in tooling
    used there

  • Following the kilo reform, the stable team is now focused on
    defining and enforcing a common stable branch policy[1], rather
    than approving every patch. Being more visible and having more
    dedicated members can only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused
    on release management and does not have the history I had with
    stable branch policy. So it might be the right moment to refocus
    release management solely on release management and get the stable
    team its own leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources
    being dedicated to it

  • If the team expands, it could finally own stable branch health
    and gate fixing. If that ends up all falling under the same roof,
    that team could make decisions on support timeframes as well,
    since it will be the primary resource to make that work

Isn't this kind of already what the stable maint team does? Well,
that and some QA people like mtreinish and sdague.

So.. good idea ? bad idea ? What do current stable-maint-core[2]
members think of that ? Who thinks they could step up to lead that
team ?

[1]
http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

With the decentralizing of the stable branch stuff in Liberty [1] it
seems like there would be less use for a PTL for stable branch
maintenance - the cats are now herding themselves, right? Or at
least that's the plan as far as I understood it. And the existing
stable branch wizards are more or less around for help and answering
questions.

The same might be said about releasing from master and the release
management team. There's still some benefit to having people dedicated
to making sure projects all agree to sane policies and to keep up with
deliverables that need to be released.

Except the distinction is that relmgt is actually producing something. Relmgt
has the releases repo which does centralize library releases, reno to do the
release notes, etc. What does the global stable core do? Right now it's there
almost entirely to just add people to the project specific stable core teams.

-Matt Treinish

I'd like to move the discussion from what are the roles of the current stable-maint-core and more towards what the benefits would be having a stable-maint team rather than the -core group alone.

Personally I think the stable maintenance should be quite a lot more than unblocking gate and approving people allowed to merge to the stable branches.

Sure, but that's not we're talking about here are we? The other tasks, like
backporting changes for example, have been taken on by project teams. Even in
your other email you mentioned that you've been doing backports and other tasks
that you consider stable maint in a glance only context. That's something we
changed in kilo which ttx referenced in [1] to enable that to happen, and it was
the only way to scale things.

The discussion here is about the cross project effort around stable branches,
which by design is a more limited scope now. Right now the cross project effort
around stable branch policy is really 2 things (both of which ttx already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

The only lever on #2 is that the global stable-maint-core is the only group
which has add permissions to the per project stable core groups. (also the
stable branch policy wiki, but that rarely changes) We specifically shrunk it to
these 2 things in. [1] Well, really 3 things there, but since we're not doing
integrated stable point releases in the future its now only 2.

This is my whole argument that creating a new team doesn't do anything. Living
under rel-mgt, some other project, or creating a new one isn't going to change
the fact that cross-project stable maint is the same 2 tasks which basically
nobody cares about, which TBH your email is just an indication of. Even if we
wanted to grow to something beyond these 2 tasks they would still be the core of
whatever it becomes and a lack of people caring about them will just block any
potential scope growth.

Frankly, this is the problem with any ML or open discussion about stable branch
maint. Everyone has an opinion about everything but lacks actual context or
never steps up to work on the cross-project side of this. I don't think creating
a new team that has no repos or other artifacts and therefore no stackalytics
credit (and by extension corporate chest thumping) is magically going to create
people actually working on this.

The logical follow on idea to the above is to create a stable branch policy doc
repo and a project team to own that. But, I still don't think that solves the
problem, especially since the real priority issue in this space is #1 which
at most is a handful of us bothering to look at that, (well that and the stable
policy barely ever changes) which honestly really isn't the majority of the
stable-maint-core group. We also have a similar problem with gate issues on
master and it's mostly the same people looking at these things there too.

IMHO the only way for creating a separate project team to be useful is to
re-centralize more responsibilities of stable maint, which is something I don't
think we should do because we'll just hit the same scaling issues we had before
kilo which prompted that change in the first place. It's the same problem every
horizontal, cross project, or whatever you want to call it effort has with
OpenStack's growth and why most have moved to the distributed self service
models.

But, if the goal is just to not make Doug responsible for #2, which is something
ttx was primarily doing before, then I guess it's fine but we should be honest
about it and just make ttx the new team leader. :) I honestly don't think
anything else discussed will change by explicitly making it a separate team.

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078
281.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Matty T got all P&V in Tokyo! :)

I pretty much agree with everything said above. I don't think we make
many policy changes, the stable maint core team is already pretty
decentralized as it is. We rarely seem to communicate much except when
the (1) gate is wedged or (2) we're coming up on a release and need a
push for reviews.

And this is perhaps one of the problems. I don't think decentralizing
the stable team is enough to have proper stable maintenance. It's a
great step forward and I don't think the proposal is to remove that.

Just like for the releases (especially for projects following
coordinated releases), stable branches could benefit from having
someone looking after them (for those projects that apply to the tag
ttx mentioned in one of his previous emails) and coordinate releases,
status, etc.

I think there's more to it than enforcing policies and keeping the
gate sane. What about projects w/o backports? Are we encouraging ppl
to do enough backports to stable branches?

The small group of gate thumpers raise hell when (1) comes up, and the
stable branch champion coordinates (2).

I've asked this in other threads before, but do we need a PTL for
this? I'm assuming ttx doesn't want this since he's asking for
volunteers. Doug doesn't want it because of release mgmt duties. But
what does a stable branch PTL do? If we have policy changes on stable
that need discussing, can't we come together and discuss those as
needed as a group?

Here's a couple of things that I'd love to see the volunteer for that
job to do:

  • Help coordinating stable releases when they are needed.
  • Help creating tools to monitor existing backports.
  • Help creating tools that could help identifying patches that might
    be worth backporting.
  • Be a contact for whenever things break in stable branches and
    coordinate with folks that can help fixing those issues.

Not sure if all the above points are worth it but I do see this as an
useful role to have.

Cheers,
Flavio

And for gate issues, the people that generally work on those are doing
it because it benefits them, either because it keeps them from having
to maintain forks out of tree or because it gets other changes landed
that are needed for their product/customers. I don't see a new team
making a difference in attracting new people to really dig into the
ugly job of unwedging the gate on stable - it takes quite a bit of
time to get up to speed. Tony is a great example of someone that has
kind of come out of left field and said he wants to help with this and
has stuck with it long enough that he makes a difference and it's
worth spending time mentoring him on the stable gate issues, but those
people are far and few between.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 11, 2015 by Flavio_Percoco (36,960 points)   3 8 11
0 votes

On 11/11/2015 8:51 AM, Flavio Percoco wrote:
On 09/11/15 21:30 -0600, Matt Riedemann wrote:

On 11/9/2015 9:12 PM, Matthew Treinish wrote:

On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:

On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:

Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:

On 11/9/2015 10:41 AM, Thierry Carrez wrote:

Hi everyone,

A few cycles ago we set up the Release Cycle Management team which
was a bit of a frankenteam of the things I happened to be leading:
release management, stable branch maintenance and vulnerability
management.
While you could argue that there was some overlap between those
functions (as in, "all these things need to be released") logic
was not the primary reason they were put together.

When the Security Team was created, the VMT was spinned out of the
Release Cycle Management team and joined there. Now I think we
should spin out stable branch maintenance as well:

  • A good chunk of the stable team work used to be stable point
    release management, but as of stable/liberty this is now done by
    the release management team and triggered by the project-specific
    stable maintenance teams, so there is no more overlap in tooling
    used there

  • Following the kilo reform, the stable team is now focused on
    defining and enforcing a common stable branch policy[1], rather
    than approving every patch. Being more visible and having more
    dedicated members can only help in that very specific mission

  • The release team is now headed by Doug Hellmann, who is focused
    on release management and does not have the history I had with
    stable branch policy. So it might be the right moment to refocus
    release management solely on release management and get the stable
    team its own leadership

  • Empowering that team to make its own decisions, giving it more
    visibility and recognition will hopefully lead to more resources
    being dedicated to it

  • If the team expands, it could finally own stable branch health
    and gate fixing. If that ends up all falling under the same roof,
    that team could make decisions on support timeframes as well,
    since it will be the primary resource to make that work

Isn't this kind of already what the stable maint team does? Well,
that and some QA people like mtreinish and sdague.

So.. good idea ? bad idea ? What do current stable-maint-core[2]
members think of that ? Who thinks they could step up to lead that
team ?

[1]
http://docs.openstack.org/project-team-guide/stable-branches.html
[2] https://review.openstack.org/#/admin/groups/530,members

With the decentralizing of the stable branch stuff in Liberty [1] it
seems like there would be less use for a PTL for stable branch
maintenance - the cats are now herding themselves, right? Or at
least that's the plan as far as I understood it. And the existing
stable branch wizards are more or less around for help and answering
questions.

The same might be said about releasing from master and the release
management team. There's still some benefit to having people
dedicated
to making sure projects all agree to sane policies and to keep up
with
deliverables that need to be released.

Except the distinction is that relmgt is actually producing
something. Relmgt
has the releases repo which does centralize library releases, reno
to do the
release notes, etc. What does the global stable core do? Right now
it's there
almost entirely to just add people to the project specific stable
core teams.

-Matt Treinish

I'd like to move the discussion from what are the roles of the
current stable-maint-core and more towards what the benefits would
be having a stable-maint team rather than the -core group alone.

Personally I think the stable maintenance should be quite a lot more
than unblocking gate and approving people allowed to merge to the
stable branches.

Sure, but that's not we're talking about here are we? The other
tasks, like
backporting changes for example, have been taken on by project teams.
Even in
your other email you mentioned that you've been doing backports and
other tasks
that you consider stable maint in a glance only context. That's
something we
changed in kilo which ttx referenced in [1] to enable that to happen,
and it was
the only way to scale things.

The discussion here is about the cross project effort around stable
branches,
which by design is a more limited scope now. Right now the cross
project effort
around stable branch policy is really 2 things (both of which ttx
already
mentioned):

  1. Keeping the gates working on the stable branches
  2. Defining and enforcing stable branch policy.

The only lever on #2 is that the global stable-maint-core is the only
group
which has add permissions to the per project stable core groups.
(also the
stable branch policy wiki, but that rarely changes) We specifically
shrunk it to
these 2 things in. [1] Well, really 3 things there, but since we're
not doing
integrated stable point releases in the future its now only 2.

This is my whole argument that creating a new team doesn't do
anything. Living
under rel-mgt, some other project, or creating a new one isn't going
to change
the fact that cross-project stable maint is the same 2 tasks which
basically
nobody cares about, which TBH your email is just an indication of.
Even if we
wanted to grow to something beyond these 2 tasks they would still be
the core of
whatever it becomes and a lack of people caring about them will just
block any
potential scope growth.

Frankly, this is the problem with any ML or open discussion about
stable branch
maint. Everyone has an opinion about everything but lacks actual
context or
never steps up to work on the cross-project side of this. I don't
think creating
a new team that has no repos or other artifacts and therefore no
stackalytics
credit (and by extension corporate chest thumping) is magically going
to create
people actually working on this.

The logical follow on idea to the above is to create a stable branch
policy doc
repo and a project team to own that. But, I still don't think that
solves the
problem, especially since the real priority issue in this space is #1
which
at most is a handful of us bothering to look at that, (well that and
the stable
policy barely ever changes) which honestly really isn't the majority
of the
stable-maint-core group. We also have a similar problem with gate
issues on
master and it's mostly the same people looking at these things there
too.

IMHO the only way for creating a separate project team to be useful
is to
re-centralize more responsibilities of stable maint, which is
something I don't
think we should do because we'll just hit the same scaling issues we
had before
kilo which prompted that change in the first place. It's the same
problem every
horizontal, cross project, or whatever you want to call it effort has
with
OpenStack's growth and why most have moved to the distributed self
service
models.

But, if the goal is just to not make Doug responsible for #2, which
is something
ttx was primarily doing before, then I guess it's fine but we should
be honest
about it and just make ttx the new team leader. :) I honestly don't
think
anything else discussed will change by explicitly making it a
separate team.

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078
281.html


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Matty T got all P&V in Tokyo! :)

I pretty much agree with everything said above. I don't think we make
many policy changes, the stable maint core team is already pretty
decentralized as it is. We rarely seem to communicate much except when
the (1) gate is wedged or (2) we're coming up on a release and need a
push for reviews.

And this is perhaps one of the problems. I don't think decentralizing
the stable team is enough to have proper stable maintenance. It's a
great step forward and I don't think the proposal is to remove that.

Just like for the releases (especially for projects following
coordinated releases), stable branches could benefit from having
someone looking after them (for those projects that apply to the tag
ttx mentioned in one of his previous emails) and coordinate releases,
status, etc.

I think there's more to it than enforcing policies and keeping the
gate sane. What about projects w/o backports? Are we encouraging ppl
to do enough backports to stable branches?

The small group of gate thumpers raise hell when (1) comes up, and the
stable branch champion coordinates (2).

I've asked this in other threads before, but do we need a PTL for
this? I'm assuming ttx doesn't want this since he's asking for
volunteers. Doug doesn't want it because of release mgmt duties. But
what does a stable branch PTL do? If we have policy changes on stable
that need discussing, can't we come together and discuss those as
needed as a group?

Here's a couple of things that I'd love to see the volunteer for that
job to do:

  • Help coordinating stable releases when they are needed.

But isn't this why we decentralized the stable release stuff? So you
don't need a single person or small team herding all the cats?

  • Help creating tools to monitor existing backports.

I'm not sure a PTL is needed for this. And I don't think a tool is
needed, we have gerrit. With decentralized stable maint to the projects
(including releasing), if I'm a nova person that cares about stable (and
I am), then I can find out what's open for stable/liberty backports
using gerrit:

  • Help creating tools that could help identifying patches that might
    be worth backporting.

Again, I'm not sure a PTL is needed for this, just some hackers. I could
see value in writing a script that scans launchpad for things like
'liberty-backport-potential' where the fix is merged in mitaka but not
proposed to stable/liberty.

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty,n,z

Easy peasy. We actually have these same dashboards linked into our
weekly nova meeting agenda just for awareness.

  • Be a contact for whenever things break in stable branches and
    coordinate with folks that can help fixing those issues.

We have the #openstack-stable channel, people can ask for help there. We
have the mailing list where stable blocking issues are regularly brought
forward to the wider audience. I guess being PTL would mean there is a
lightning rod when something goes bad, but I'd think the majority of the
time they are going to defer to the PTL of whatever project is blocked
and get them to work on it, i.e. if trove stable/liberty is blocked b/c
of some issue in their own CI tooling.

Not sure if all the above points are worth it but I do see this as an
useful role to have.

Cheers,
Flavio

And for gate issues, the people that generally work on those are doing
it because it benefits them, either because it keeps them from having
to maintain forks out of tree or because it gets other changes landed
that are needed for their product/customers. I don't see a new team
making a difference in attracting new people to really dig into the
ugly job of unwedging the gate on stable - it takes quite a bit of
time to get up to speed. Tony is a great example of someone that has
kind of come out of left field and said he wants to help with this and
has stuck with it long enough that he makes a difference and it's
worth spending time mentoring him on the stable gate issues, but those
people are far and few between.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 11, 2015 by Matt_Riedemann (48,320 points)   3 9 22
...