settingsLogin | Registersettings

[openstack-dev] Upstream LTS Releases

0 votes

Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Nov 15, 2017 in openstack-operators by Erik_McCormick (3,880 points)   2 3

22 Responses

0 votes

Erik McCormick emccormick@cirrusseven.com writes:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

I regret that due to a conflict I was unable to attend this session.
Can you elaborate on why third-party CI would be necessary for this,
considering that upstream CI already exists on all active branches?

Thanks,

Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 7, 2017 by James_E._Blair (5,080 points)   1 4 5
0 votes

On Tue, Nov 7, 2017 at 6:45 PM, James E. Blair corvus@inaugust.com wrote:
Erik McCormick emccormick@cirrusseven.com writes:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

I regret that due to a conflict I was unable to attend this session.
Can you elaborate on why third-party CI would be necessary for this,
considering that upstream CI already exists on all active branches?

Thanks,

Jim

Lack of infra resources, people are already maintaining their own
testing for old releases, and distribution of work across
organizations I think were the chief reasons. Someone else feel free
to chime in and expand on it.

-Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 7, 2017 by Erik_McCormick (3,880 points)   2 3
0 votes

Erik McCormick wrote:
This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

I took the action of summarizing the discussion in more detail, will do
as soon as my brain is not as mushy, which might take a couple of weeks :)

Note that it's not really about devs vs. ops, with devs abdicating all
responsibility on stable branches : it's about allowing collaboration on
patches beyond EOL (which is what we are able to support with "live"
stable branches on evolving OS/PyPI substrate) and enable whoever steps
up to maintain longer-lived branches to come up with a set of tests that
actually match their needs (tests that would be less likely to bitrot
due to changing OS/PyPI substrate).

A number of people from all backgrounds volunteered to flesh out a more
detailed proposal. Watch that space!

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Thierry_Carrez (57,480 points)   3 7 12
0 votes

Erik McCormick emccormick@cirrusseven.com writes:

On Tue, Nov 7, 2017 at 6:45 PM, James E. Blair corvus@inaugust.com wrote:

Erik McCormick emccormick@cirrusseven.com writes:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

I regret that due to a conflict I was unable to attend this session.
Can you elaborate on why third-party CI would be necessary for this,
considering that upstream CI already exists on all active branches?

Lack of infra resources, people are already maintaining their own
testing for old releases, and distribution of work across
organizations I think were the chief reasons. Someone else feel free
to chime in and expand on it.

Which resources are lacking? I wasn't made aware of a shortage of
upstream CI resources affecting stable branch work, but if there is, I'm
sure we can address it -- this is a very important effort.

The upstream CI system is also a collaboratively maintained system with
folks from many organizations participating in it. Indeed we're now
distributing its maintenance and operation into projects themselves.
It seems like an ideal place for folks from different organizations to
collaborate.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by James_E._Blair (5,080 points)   1 4 5
0 votes

On Nov 8, 2017 1:52 PM, "James E. Blair" corvus@inaugust.com wrote:

Erik McCormick emccormick@cirrusseven.com writes:

On Tue, Nov 7, 2017 at 6:45 PM, James E. Blair corvus@inaugust.com
wrote:

Erik McCormick emccormick@cirrusseven.com writes:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

I regret that due to a conflict I was unable to attend this session.
Can you elaborate on why third-party CI would be necessary for this,
considering that upstream CI already exists on all active branches?

Lack of infra resources, people are already maintaining their own
testing for old releases, and distribution of work across
organizations I think were the chief reasons. Someone else feel free
to chime in and expand on it.

Which resources are lacking? I wasn't made aware of a shortage of
upstream CI resources affecting stable branch work, but if there is, I'm
sure we can address it -- this is a very important effort.

It's not a matter of things lacking for today's release cadence and
deprecation policy. That is working fine. The problems would come if you
had to, say, continue to run it for Mitaka until Queens is released.

The upstream CI system is also a collaboratively maintained system with
folks from many organizations participating in it. Indeed we're now
distributing its maintenance and operation into projects themselves.
It seems like an ideal place for folks from different organizations to
collaborate.

Monty, as well as the Stable Branch cores, were in the room, so perhaps
they can elaborate on this for us. I'm no expert on what can and cannot be
done.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by Erik_McCormick (3,880 points)   2 3
0 votes

On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
emccormick@cirrusseven.com wrote:
Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

In advance, pardon the defensive tone. I was not in a position to
attend, or even be in Sydney. However, as this comes across the ML, I
can't help but get the impression this effort would be forcing more
work on already stretched teams, ie. deployment-focused development
teams already under a crunch as contributor count continues to decline
in favor of other projects inside and out of OpenStack.

As a friendly reminder, Chef is still actively developed, though we've
not had a great return from recruiting more people. We have about 3.5
active developers, including active cores, and non-cores who felt it
worthwhile to contribute back upstream. There is no major corporate
backer here, but merely a handful of potentially stubborn volunteers.
Nobody is behind the curtain, but Chef OpenStack still have a few
active users (once again, I point to the annual User Survey results)
and contributors. However, we do not use the MLs as a primary
communication means, so I can see how we might be forgotten or
ignored.

In practice, no one likes talking about Chef OpenStack that I've
experienced, neither in the Chef or OpenStack communities. However, as
a maintainer, I keep making it a point to bring it up when it seems
the project gets papered over, or the core team gets signed up for
more work decided in a room half a world away. Admittedly, the whole
deployment method is a hard sell if you're not using Chef in some way.
It has always been my takeaway that the project was merely tolerated
under the OpenStack designation, neither embraced nor even liked, even
being the "official" OpenStack deployment method for a major
deployment toolset. The Foundation's support has been outstanding when
we've needed it, but that's about as far as the delightful goes. The
Chef community is a bit more tolerant of someone using the Chef
moniker for OpenStack, but migrating from Gerrit to GitHub is a major
undertaking that the development team may or may not be able to
reasonably support without more volunteers. Now that the proposition
exists about making a Stable Release liaison derived from existing
cores, I can't help but get the impression that, for active-but-quiet
projects, it'll be yet another PTL responsibility to keep up with, in
addition to the rigors that already come with the role. I'm hoping
I'll be proven wrong here, but I can and do get in trouble for hoping.

--
Best,
Samuel Cassiba


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 8, 2017 by s_at_cassiba.com (1,200 points)  
0 votes

Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:

On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
emccormick@cirrusseven.com wrote:

Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

In advance, pardon the defensive tone. I was not in a position to
attend, or even be in Sydney. However, as this comes across the ML, I
can't help but get the impression this effort would be forcing more
work on already stretched teams, ie. deployment-focused development
teams already under a crunch as contributor count continues to decline
in favor of other projects inside and out of OpenStack.

I suspect if LTS's become a normal part of OpenStack, most deployment
projects will decline to support the interim releases. We can infer this
from the way Ubuntu is used. This might actually be a good thing for the
chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
and the 0.5 person can do some best effort to cover the weird corner
case of "previous stable release to master".

The biggest challenge will be ensuring that the skip-level upgrades
work. The current grenade based upgrade jobs are already quite a bear to
keep working IIRC. I've not seen if chef or any of the deployment projects
test upgrades like that.

However, if people can stop caring much about the interim releases and
just keep "previous LTS to master" upgrades working, then that might be
good for casual adoption.

Personally I'd rather we make it easier to run "rolling release"
OpenStack. Maybe we can do that if we stop cutting stable releases every
6 months.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 9, 2017 by Clint_Byrum (40,940 points)   4 5 6
0 votes

On 2017-11-08 23:15:15 -0800 (-0800), Clint Byrum wrote:
[...]
The biggest challenge will be ensuring that the skip-level upgrades
work. The current grenade based upgrade jobs are already quite a bear to
keep working IIRC. I've not seen if chef or any of the deployment projects
test upgrades like that.
[...]

Another challenge which has been mostly hand-waved away at this
stage is the distro support piece. Queens is being tested on Ubuntu
16.04 but our "S" release will likely be tested on Ubuntu 18.04
instead... so effective testing for a skip-level upgrade between
those two LTS releases will also involve in-place upgrading of
Ubuntu.
--
Jeremy Stanley


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 9, 2017 by Jeremy_Stanley (56,700 points)   3 4 7
0 votes

On Thu, Nov 09, 2017 at 04:34:24PM +0000, Jeremy Stanley wrote:
:On 2017-11-08 23:15:15 -0800 (-0800), Clint Byrum wrote:
:[...]
:> The biggest challenge will be ensuring that the skip-level upgrades
:> work. The current grenade based upgrade jobs are already quite a bear to
:> keep working IIRC. I've not seen if chef or any of the deployment projects
:> test upgrades like that.
:[...]
:
:Another challenge which has been mostly hand-waved away at this
:stage is the distro support piece. Queens is being tested on Ubuntu
:16.04 but our "S" release will likely be tested on Ubuntu 18.04
:instead... so effective testing for a skip-level upgrade between
:those two LTS releases will also involve in-place upgrading of
:Ubuntu.

Having done inplace Ubuntu upgrades from 12.04->14.04->16.04
underneath "production"* openstack I'm not too worried about the
mechanics of that.

Currently for Ubuntu case when you get to a release boundary you need
to bring old Distro Release upto Newest OpenStack then move to new
Distro Release.

You can push production load to another node, reinstall new version of
Ubuntu with current configs then move load back. This does make some
assumptions about the deployed architecture but hopefully if nonstop
cloud is what you're after you've deployed something that's resilient
to single node faults which is basically what a Distro upgrade takes.

-Jon

:--
:Jeremy Stanley
:
:_______________________________________________
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 9, 2017 by jon_at_csail.mit.edu (4,720 points)   1 2 5
0 votes

Excerpts from Clint Byrum's message of 2017-11-08 23:15:15 -0800:

Excerpts from Samuel Cassiba's message of 2017-11-08 08:27:12 -0800:

On Tue, Nov 7, 2017 at 3:28 PM, Erik McCormick
emccormick@cirrusseven.com wrote:

Hello Ops folks,

This morning at the Sydney Summit we had a very well attended and very
productive session about how to go about keeping a selection of past
releases available and maintained for a longer period of time (LTS).

There was agreement in the room that this could be accomplished by
moving the responsibility for those releases from the Stable Branch
team down to those who are already creating and testing patches for
old releases: The distros, deployers, and operators.

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.

Please take a look at the Etherpad from the session if you'd like to
see the details. More importantly, if you would like to contribute to
this effort, please add your name to the list starting on line 133.

https://etherpad.openstack.org/p/SYD-forum-upstream-lts-releases

Thanks to everyone who participated!

Cheers,
Erik


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

In advance, pardon the defensive tone. I was not in a position to
attend, or even be in Sydney. However, as this comes across the ML, I
can't help but get the impression this effort would be forcing more
work on already stretched teams, ie. deployment-focused development
teams already under a crunch as contributor count continues to decline
in favor of other projects inside and out of OpenStack.

I suspect if LTS's become a normal part of OpenStack, most deployment
projects will decline to support the interim releases. We can infer this
from the way Ubuntu is used. This might actually be a good thing for the
chef OpenStack community. 3 out of 3.5 of you can focus on the LTS bits,
and the 0.5 person can do some best effort to cover the weird corner
case of "previous stable release to master".

The biggest challenge will be ensuring that the skip-level upgrades
work. The current grenade based upgrade jobs are already quite a bear to
keep working IIRC. I've not seen if chef or any of the deployment projects
test upgrades like that.

However, if people can stop caring much about the interim releases and
just keep "previous LTS to master" upgrades working, then that might be
good for casual adoption.

Personally I'd rather we make it easier to run "rolling release"
OpenStack. Maybe we can do that if we stop cutting stable releases every
6 months.

We should stop calling what we're talking about "LTS". It isn't
going to match the expectations of anyone receiving LTS releases
for other products, at least at first. Perhaps "Deployer Supported"
or "User Supported" are better terms for what we're talking about.

In the "LTS" room we did not agree to stop cutting stable releases
or to start supporting upgrades directly from N-2 (or older) to N.
Both of those changes would require modifying the support the
existing contributor base has committed to provide.

Fast-forward upgrades will still need to run the migration steps
of each release in order, one at a time. The team working on that
is going to produce a document describing what works today so we
can analyze it for ways to improve the upgrade experience, for both
fast-forward and "regular" upgrades. That was all discussed in a
separate session.

Doug


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 10, 2017 by Doug_Hellmann (87,520 points)   3 4 4
...