settingsLogin | Registersettings

Re: [openstack-dev] Upstream LTS Releases

0 votes

Hi!

This is amazing to see this discussed! Looking forward to more details.

On 11/08/2017 12:28 AM, Erik McCormick wrote:

Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

What is the most worrying is the exact "take over" process. Does it mean that
the teams will give away the +2 power to a different team? Or will our (small)
stable teams still be responsible for landing changes? If so, will they have to
learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the control over
quality at the same time :) Probably the final proposal will clarify it..

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 15, 2017 in openstack-dev by Dmitry_Tantsur (18,080 points)   2 3 7

39 Responses

0 votes

On 11/14/2017 05:08 PM, Bogdan Dobrelya wrote:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

What is the most worrying is the exact "take over" process. Does it mean that
the teams will give away the +2 power to a different team? Or will our (small)
stable teams still be responsible for landing changes? If so, will they have
to learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the control
over quality at the same time :) Probably the final proposal will clarify it..

The quality of backported fixes is expected to be a direct (and only?) interest
of those new teams of new cores, coming from users and operators and vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.

The more parties to establish their 3rd party checking jobs, the better proposed
changes communicated, which directly affects the quality in the end. I also
suppose, contributors from ops world will likely be only struggling to see
things getting fixed, and not new features adopted by legacy deployments they're
used to maintain. So in theory, this works and as a mainstream developer and
maintainer, you need no to fear of losing control over LTS code :)

Another question is how to not block all on each over, and not push contributors
away when things are getting awry, jobs failing and merging is blocked for a
long time, or there is no consensus reached in a code review. I propose the LTS
policy to enforce CI jobs be non-voting, as a first step on that way, and giving
every LTS team member a core rights maybe? Not sure if that works though.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Dmitry_Tantsur (18,080 points)   2 3 7
0 votes

On 11/14/2017 06:21 PM, Erik McCormick wrote:
On Tue, Nov 14, 2017 at 11:30 AM, Blair Bethwaite
blair.bethwaite@gmail.com wrote:

Hi all - please note this conversation has been split variously across
-dev and -operators.

One small observation from the discussion so far is that it seems as
though there are two issues being discussed under the one banner:
1) maintain old releases for longer
2) do stable releases less frequently

It would be interesting to understand if the people who want longer
maintenance windows would be helped by #2.

I would like to hear from people who do not want #2 and why not.
What are the benefits of 6 months vs. 1 year? I have heard objections
in the hallway track, but I have struggled to retain the rationale for
more than 10 seconds. I think this may be more of a religious
discussion that could take a while though.

One point is maintenance burden. Everything that has to be deprecated and
removed will have to be kept for twice more time in the worst case.

The second point is that contributors, from my experience, don't like waiting
many months for their shiny feature to get released. That will increase pressure
on the teams in the end of every release to get everything in - or it will have
to wait 1 year.

Note that both points apply even if you do "less-stable" releases between stable
ones.

1 is something we can act on right now with the eventual goal of

being able to skip releases entirely. We are addressing the
maintenance of the old issue right now. As we get farther down the
road of fast-forward upgrade tooling, then we will be able to please
those wishing for a slower upgrade cadence, and those that want to
stay on the bleeding edge simultaneously.

-Erik

On 14 November 2017 at 09:25, Doug Hellmann doug@doughellmann.com wrote:

Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

What is the most worrying is the exact "take over" process. Does it mean that
the teams will give away the +2 power to a different team? Or will our (small)
stable teams still be responsible for landing changes? If so, will they have to
learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the control over
quality at the same time :) Probably the final proposal will clarify it..

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and
operators and vendors. The more parties to establish their 3rd party

We have an unhealthy focus on "3rd party" jobs in this discussion. We
should not assume that they are needed or will be present. They may be,
but we shouldn't build policy around the assumption that they will. Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?

checking jobs, the better proposed changes communicated, which directly
affects the quality in the end. I also suppose, contributors from ops
world will likely be only struggling to see things getting fixed, and
not new features adopted by legacy deployments they're used to maintain.
So in theory, this works and as a mainstream developer and maintainer,
you need no to fear of losing control over LTS code :)

Another question is how to not block all on each over, and not push
contributors away when things are getting awry, jobs failing and merging
is blocked for a long time, or there is no consensus reached in a code
review. I propose the LTS policy to enforce CI jobs be non-voting, as a
first step on that way, and giving every LTS team member a core rights
maybe? Not sure if that works though.

I'm not sure what change you're proposing for CI jobs and their voting
status. Do you mean we should make the jobs non-voting as soon as the
branch passes out of the stable support period?

Regarding the review team, anyone on the review team for a branch
that goes out of stable support will need to have +2 rights in that
branch. Otherwise there's no point in saying that they're maintaining
the branch.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Cheers,
~Blairo


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Dmitry_Tantsur (18,080 points)   2 3 7
0 votes

On Tue, Nov 14, 2017 at 11:28 AM, Dmitry Tantsur dtantsur@redhat.com wrote:
On 11/14/2017 05:08 PM, Bogdan Dobrelya wrote:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

What is the most worrying is the exact "take over" process. Does it mean
that the teams will give away the +2 power to a different team? Or will our
(small) stable teams still be responsible for landing changes? If so, will
they have to learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the
control over quality at the same time :) Probably the final proposal will
clarify it..

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators
and vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved
in a decision whether to make a backport or not. Will these people be able
to evaluate a risk of each patch? Do they have enough context on how that
release was implemented and what can break? Do they understand why feature
backports are bad? Why they should not skip (supported) releases when
backporting?

I know a lot of very reasonable people who do not understand the things
above really well.

I think there is more of a general "yes, but..." feel and not so much
a misunderstanding or lack of understanding entirely. With my operator
and PTL hats on, I'm in favor of a release cadence that is favorable
for the people involved. It's already proven that the current model
is broken or lacking in some way, simply by having these
conversations. With the status quo, it's almost a death march from one
release to the next, but nobody really wants to prolong that pain
because this topic comes up again and again.

Ideally, contributors are empowered enough to pick up the reins and
deliver the changes themselves, and some are, but it's pretty damned
daunting from the outside. The new contributors who want to contribute
but don't see the way in, probably because we haven't said mellon, are
left scratching their heads and eventually deem OpenStack as Not
Ready. It's almost like a perception exists that being able to even
submit a one-line patch is a gate to admittance. Unfortunately, less
and less are willing to pay that toll, no matter how nice the project
is on the other side.

--
Best,
Samuel Cassiba


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by s_at_cassiba.com (1,200 points)  
0 votes

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.

I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

On Tue, Nov 14, 2017 at 11:25:03AM -0500, Doug Hellmann wrote:
Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

What is the most worrying is the exact "take over" process. Does it mean that
the teams will give away the +2 power to a different team? Or will our (small)
stable teams still be responsible for landing changes? If so, will they have to
learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the control over
quality at the same time :) Probably the final proposal will clarify it..

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and
operators and vendors. The more parties to establish their 3rd party

We have an unhealthy focus on "3rd party" jobs in this discussion. We
should not assume that they are needed or will be present. They may be,
but we shouldn't build policy around the assumption that they will. Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?

I get the feeling more people are comfortable contributing to their own 3rd
party CI then upstream openstack CI systems. Either because they don't have the
time or don't understand how it works. I agree with Doug, for this to work, I
think we need to have a health amount of people helping keep the CI systems
running upstream, then depending on 3rd party CI downstream.

As a comment, I think getting involved in the openstack-stablemaint team will be
a good first step towards the goals people are interested in here. I'm happy to
help work with others, and I'm taking tonyb up on his offer to start helping too
:)

checking jobs, the better proposed changes communicated, which directly
affects the quality in the end. I also suppose, contributors from ops
world will likely be only struggling to see things getting fixed, and
not new features adopted by legacy deployments they're used to maintain.
So in theory, this works and as a mainstream developer and maintainer,
you need no to fear of losing control over LTS code :)

Another question is how to not block all on each over, and not push
contributors away when things are getting awry, jobs failing and merging
is blocked for a long time, or there is no consensus reached in a code
review. I propose the LTS policy to enforce CI jobs be non-voting, as a
first step on that way, and giving every LTS team member a core rights
maybe? Not sure if that works though.

I'm not sure what change you're proposing for CI jobs and their voting
status. Do you mean we should make the jobs non-voting as soon as the
branch passes out of the stable support period?

Regarding the review team, anyone on the review team for a branch
that goes out of stable support will need to have +2 rights in that
branch. Otherwise there's no point in saying that they're maintaining
the branch.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by pabelanger_at_redhat (6,560 points)   1 1 2
0 votes

On Wed, Nov 15, 2017 at 7:01 AM, Chris Friesen
chris.friesen@windriver.com wrote:
On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators
and
vendors.

I'm not assuming bad intentions, not at all. But there is a lot of
involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that
release
was implemented and what can break? Do they understand why feature
backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things
above
really well.

I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within
the various distros.

+100000000 Chris!

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Davanum_Srinivas (35,920 points)   2 4 8
0 votes

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.

I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris

Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Doug_Hellmann (87,520 points)   3 4 8
0 votes

Folks,

This discussion and the people interested in it seem like a perfect application of the SIG process. By turning LTS into a SIG, everyone can discuss the issues on the SIG mailing list and the discussion shouldn't end up split. If it turns into a project, great. If a solution is found that doesn't need a new project, great. Even once there is a decision on how to move forward, there will still be implementation issues and enhancements, so the SIG could very well be long-lived. But the important aspect of this is: keeping the discussion in a place where both devs and ops can follow the whole thing and act on recommendations.

Food for thought.

--Rocky

-----Original Message-----
From: Blair Bethwaite [mailto:blair.bethwaite@gmail.com]
Sent: Tuesday, November 14, 2017 8:31 AM
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org; openstack-oper.
Subject: Re: [openstack-dev] Upstream LTS Releases

Hi all - please note this conversation has been split variously across -dev and -
operators.

One small observation from the discussion so far is that it seems as though
there are two issues being discussed under the one banner:
1) maintain old releases for longer
2) do stable releases less frequently

It would be interesting to understand if the people who want longer
maintenance windows would be helped by #2.

On 14 November 2017 at 09:25, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:

The concept, in general, is to create a new set of cores from
these groups, and use 3rd party CI to validate patches. There are
lots of details to be worked out yet, but our amazing UC (User
Committee) will be begin working out the details.

What is the most worrying is the exact "take over" process. Does it
mean that the teams will give away the +2 power to a different
team? Or will our (small) stable teams still be responsible for
landing changes? If so, will they have to learn how to debug 3rd party CI
jobs?

Generally, I'm scared of both overloading the teams and losing the
control over quality at the same time :) Probably the final proposal will
clarify it..

The quality of backported fixes is expected to be a direct (and
only?) interest of those new teams of new cores, coming from users
and operators and vendors. The more parties to establish their 3rd
party

We have an unhealthy focus on "3rd party" jobs in this discussion. We
should not assume that they are needed or will be present. They may
be, but we shouldn't build policy around the assumption that they
will. Why would we have third-party jobs on an old branch that we
don't have on master, for instance?

checking jobs, the better proposed changes communicated, which
directly affects the quality in the end. I also suppose, contributors
from ops world will likely be only struggling to see things getting
fixed, and not new features adopted by legacy deployments they're used
to maintain.
So in theory, this works and as a mainstream developer and
maintainer, you need no to fear of losing control over LTS code :)

Another question is how to not block all on each over, and not push
contributors away when things are getting awry, jobs failing and
merging is blocked for a long time, or there is no consensus reached
in a code review. I propose the LTS policy to enforce CI jobs be
non-voting, as a first step on that way, and giving every LTS team
member a core rights maybe? Not sure if that works though.

I'm not sure what change you're proposing for CI jobs and their voting
status. Do you mean we should make the jobs non-voting as soon as the
branch passes out of the stable support period?

Regarding the review team, anyone on the review team for a branch that
goes out of stable support will need to have +2 rights in that branch.
Otherwise there's no point in saying that they're maintaining the
branch.

Doug



____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Cheers,
~Blairo



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-
request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Rochelle_Grober (7,040 points)   1 3 3
0 votes

On 11/14/2017 02:10 PM, Doug Hellmann wrote:
Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.

I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris

Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?

For what it's worth, we often lag more than 6 months behind master and so some
of the things we backport wouldn't be allowed by the existing stable branch
support phases. (ie aren't "critical" or security patches.)

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

Excerpts from Chris Friesen's message of 2017-11-14 15:50:08 -0600:

On 11/14/2017 02:10 PM, Doug Hellmann wrote:

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:

The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.

I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.

I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris

Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?

For what it's worth, we often lag more than 6 months behind master and so some
of the things we backport wouldn't be allowed by the existing stable branch
support phases. (ie aren't "critical" or security patches.)

Chris

We should include a review of some of those policies as part of
this discussion. It would seem very odd to have a fix land in master,
not make it into the stable branches, and then show up in a branch
following the LTS policy.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 14, 2017 by Doug_Hellmann (87,520 points)   3 4 8
...