settingsLogin | Registersettings

[openstack-dev] [release][requirements][packaging][summit] input needed on summit discussion about global requirements

0 votes

I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 17, 2016 in openstack-dev by Doug_Hellmann (87,520 points)   3 4 7

72 Responses

0 votes

On 04/17/2016 10:13 AM, Doug Hellmann wrote:
I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements

Sadly the session conflicts with a different one that I'm leading, so I
cannot be there. That, of course, makes me sad, because I think it's an
important conversation to have, and I have some strong opinions on the
topic in both directions.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 17, 2016 by Monty_Taylor (22,780 points)   2 4 7
0 votes

Excerpts from Monty Taylor's message of 2016-04-17 10:34:36 -0500:

On 04/17/2016 10:13 AM, Doug Hellmann wrote:

I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements

Sadly the session conflicts with a different one that I'm leading, so I
cannot be there. That, of course, makes me sad, because I think it's an
important conversation to have, and I have some strong opinions on the
topic in both directions.

Monty

Bummer. Please feel free to add any comments to the etherpad and I'll
try to proxy you.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 17, 2016 by Doug_Hellmann (87,520 points)   3 4 7
0 votes

On 04/17/2016 10:34 AM, Monty Taylor wrote:
On 04/17/2016 10:13 AM, Doug Hellmann wrote:

I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1]
https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements

Sadly the session conflicts with a different one that I'm leading, so I
cannot be there. That, of course, makes me sad, because I think it's an
important conversation to have, and I have some strong opinions on the
topic in both directions.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
If you will be in the other cross project sessions we can talk, we might
(or might not) share similar opinions :P

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 17, 2016 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

Monty Taylor wrote:
On 04/17/2016 10:13 AM, Doug Hellmann wrote:

I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1]
https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements

Sadly the session conflicts with a different one that I'm leading, so I
cannot be there. That, of course, makes me sad, because I think it's an
important conversation to have, and I have some strong opinions on the
topic in both directions.

We might be able to adapt the schedule to accommodate your presence...
if we do the change ASAP and communicate it widely.

We could for example swap the "Co-installability Requirements"
discussion with the "Stable Branch End of Life Policy" discussion.

Such a swap could help with the conflict someone reported between
"Identity v3 API only devstack" and the "Stable Branch" discussion
(can't remember who/where though).

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2016 by Thierry_Carrez (57,480 points)   3 8 12
0 votes

On 04/17/2016 11:34 AM, Monty Taylor wrote:
On 04/17/2016 10:13 AM, Doug Hellmann wrote:

I am organizing a summit session for the cross-project track to
(re)consider how we manage our list of global dependencies [1].
Some of the changes I propose would have a big impact, and so I
want to ensure everyone doing packaging work for distros is available
for the discussion. Please review the etherpad [2] and pass the
information along to colleagues who might be interested.

Doug

[1]
https://www.openstack.org/summit/austin-2016/summit-schedule/events/9473
[2] https://etherpad.openstack.org/p/newton-global-requirements

Sadly the session conflicts with a different one that I'm leading, so I
cannot be there. That, of course, makes me sad, because I think it's an
important conversation to have, and I have some strong opinions on the
topic in both directions.

Whether or not this session gets moved around to accommodate conflicts,
this session represents potentially the most disruptive change up for
consideration this cycle, which means this is going to have to include a
community conversation beyond just the DS session.

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On Mon, 18 Apr 2016, Sean Dague wrote:

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

I won't be at summit and I feel pretty strongly about this topic, so
I'll throw out my comments:

I agree with the basic premise: In the big tent universe co-
installability is holding us back and is a huge cost in terms of spent
energy. In a world where service isolation is desirable and common
(whether by virtualenv, containers, different hosts, etc) targeting an
all-in-one install seems only to serve the purposes of all-in-one rpm-
or deb-based installations.

Many (most?) people won't be doing those kinds of installations. If all-in-
one installations are important to the rpm- and deb- based distributions
then they should be resolving the dependency issues local to their own
infrastructure (or realizing that it is too painful and start
containerizing or otherwise as well).

I think making these changes will help to improve and strengthen the
boundaries and contracts between services. If not technically then
at least socially, in the sense that the negotiations that people
make to get things to work are about what actually matters in their
services, not unwinding python dependencies and the like.

A lot of the basics of getting this to work are already in place
in devstack. One challenge I've run into the past is when devstack
plugin A has made an assumption about having access to a python
script provided by devstack plugin B, but it's not on $PATH or its
dependencies are not in the site-packages visible to the current
context. The solution here is to use full paths into virtenvs.

--
Chris Dent (╯°□°)╯︵┻━┻ http://anticdent.org/
freenode: cdent tw: @anticdent__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Apr 18, 2016 by cdent_plus_os_at_ant (12,800 points)   2 2 4
0 votes

On 04/18/2016 08:22 AM, Chris Dent wrote:
On Mon, 18 Apr 2016, Sean Dague wrote:

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

I won't be at summit and I feel pretty strongly about this topic, so
I'll throw out my comments:

I agree with the basic premise: In the big tent universe co-
installability is holding us back and is a huge cost in terms of spent
energy. In a world where service isolation is desirable and common
(whether by virtualenv, containers, different hosts, etc) targeting an
all-in-one install seems only to serve the purposes of all-in-one rpm-
or deb-based installations.

Many (most?) people won't be doing those kinds of installations. If all-in-
one installations are important to the rpm- and deb- based distributions
then they should be resolving the dependency issues local to their own
infrastructure (or realizing that it is too painful and start
containerizing or otherwise as well).

I think making these changes will help to improve and strengthen the
boundaries and contracts between services. If not technically then
at least socially, in the sense that the negotiations that people
make to get things to work are about what actually matters in their
services, not unwinding python dependencies and the like.

A lot of the basics of getting this to work are already in place in
devstack. One challenge I've run into the past is when devstack
plugin A has made an assumption about having access to a python
script provided by devstack plugin B, but it's not on $PATH or its
dependencies are not in the site-packages visible to the current
context. The solution here is to use full paths into virtenvs.

As Chris said, doing virtualenvs on the Devstack side for services is
pretty much there. The team looked at doing this last year, then stopped
due to operator feedback.

One of the things that gets a little weird (when using devstack for
development) is if you actually want to see the impact of library
changes on the environment. As you'll need to make sure you loop and
install those libraries into every venv where they are used. This
forward reference doesn't really exist. So some tooling there will be
needed.

Middleware that's pushed from one project into another (like Ceilometer
-> Swift) is also a funny edge case that I think get funnier here.

Those are mostly implementation details, that probably have work
arounds, but would need people on them.

From a strategic perspective this would basically make traditional Linux
Packaging of OpenStack a lot harder. That might be the right call,
because traditional Linux Packaging definitely suffers from the fact
that everything on a host needs to be upgraded at the same time. For
large installs of OpenStack (especially public cloud cases) traditional
packages are definitely less used.

However Linux Packaging is how a lot of people get exposed to software.
The power of onboarding with apt-get / yum install is a big one.

I've been through the ups and downs of both approaches so many times now
in my own head, I no longer have a strong preference beyond the fact
that we do one approach today, and doing a different one is effort to
make the transition.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 18/04/2016 13:51, Sean Dague wrote:
On 04/18/2016 08:22 AM, Chris Dent wrote:

On Mon, 18 Apr 2016, Sean Dague wrote:

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

I won't be at summit and I feel pretty strongly about this topic, so
I'll throw out my comments:

I agree with the basic premise: In the big tent universe co-
installability is holding us back and is a huge cost in terms of spent
energy. In a world where service isolation is desirable and common
(whether by virtualenv, containers, different hosts, etc) targeting an
all-in-one install seems only to serve the purposes of all-in-one rpm-
or deb-based installations.

Many (most?) people won't be doing those kinds of installations. If all-in-
one installations are important to the rpm- and deb- based distributions
then they should be resolving the dependency issues local to their own
infrastructure (or realizing that it is too painful and start
containerizing or otherwise as well).

I think making these changes will help to improve and strengthen the
boundaries and contracts between services. If not technically then
at least socially, in the sense that the negotiations that people
make to get things to work are about what actually matters in their
services, not unwinding python dependencies and the like.

A lot of the basics of getting this to work are already in place in
devstack. One challenge I've run into the past is when devstack
plugin A has made an assumption about having access to a python
script provided by devstack plugin B, but it's not on $PATH or its
dependencies are not in the site-packages visible to the current
context. The solution here is to use full paths into virtenvs.

As Chris said, doing virtualenvs on the Devstack side for services is
pretty much there. The team looked at doing this last year, then stopped
due to operator feedback.

One of the things that gets a little weird (when using devstack for
development) is if you actually want to see the impact of library
changes on the environment. As you'll need to make sure you loop and
install those libraries into every venv where they are used. This
forward reference doesn't really exist. So some tooling there will be
needed.

Middleware that's pushed from one project into another (like Ceilometer
-> Swift) is also a funny edge case that I think get funnier here.

Those are mostly implementation details, that probably have work
arounds, but would need people on them.

From a strategic perspective this would basically make traditional Linux
Packaging of OpenStack a lot harder. That might be the right call,
because traditional Linux Packaging definitely suffers from the fact
that everything on a host needs to be upgraded at the same time. For
large installs of OpenStack (especially public cloud cases) traditional
packages are definitely less used.

However Linux Packaging is how a lot of people get exposed to software.
The power of onboarding with apt-get / yum install is a big one.

I've been through the ups and downs of both approaches so many times now
in my own head, I no longer have a strong preference beyond the fact
that we do one approach today, and doing a different one is effort to
make the transition.

-Sean

It is also worth noting that according to the OpenStack User Survey [0]
56% of deployments use "Unmodifed packages from the operating system".

Granted it was a small sample size (302 responses to that question)
but it is worth keeping this in mind as we talk about moving the burden
to packagers.

0 -
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf (page
36)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2016 by graham.hayes_at_hpe. (6,780 points)   1 2 2
0 votes

On 04/18/2016 08:24 AM, Hayes, Graham wrote:
On 18/04/2016 13:51, Sean Dague wrote:

On 04/18/2016 08:22 AM, Chris Dent wrote:

On Mon, 18 Apr 2016, Sean Dague wrote:

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

I won't be at summit and I feel pretty strongly about this topic, so
I'll throw out my comments:

I agree with the basic premise: In the big tent universe co-
installability is holding us back and is a huge cost in terms of spent
energy. In a world where service isolation is desirable and common
(whether by virtualenv, containers, different hosts, etc) targeting an
all-in-one install seems only to serve the purposes of all-in-one rpm-
or deb-based installations.

Many (most?) people won't be doing those kinds of installations. If all-in-
one installations are important to the rpm- and deb- based distributions
then they should be resolving the dependency issues local to their own
infrastructure (or realizing that it is too painful and start
containerizing or otherwise as well).

I think making these changes will help to improve and strengthen the
boundaries and contracts between services. If not technically then
at least socially, in the sense that the negotiations that people
make to get things to work are about what actually matters in their
services, not unwinding python dependencies and the like.

A lot of the basics of getting this to work are already in place in
devstack. One challenge I've run into the past is when devstack
plugin A has made an assumption about having access to a python
script provided by devstack plugin B, but it's not on $PATH or its
dependencies are not in the site-packages visible to the current
context. The solution here is to use full paths into virtenvs.

As Chris said, doing virtualenvs on the Devstack side for services is
pretty much there. The team looked at doing this last year, then stopped
due to operator feedback.

One of the things that gets a little weird (when using devstack for
development) is if you actually want to see the impact of library
changes on the environment. As you'll need to make sure you loop and
install those libraries into every venv where they are used. This
forward reference doesn't really exist. So some tooling there will be
needed.

Middleware that's pushed from one project into another (like Ceilometer
-> Swift) is also a funny edge case that I think get funnier here.

Those are mostly implementation details, that probably have work
arounds, but would need people on them.

From a strategic perspective this would basically make traditional Linux
Packaging of OpenStack a lot harder. That might be the right call,
because traditional Linux Packaging definitely suffers from the fact
that everything on a host needs to be upgraded at the same time. For
large installs of OpenStack (especially public cloud cases) traditional
packages are definitely less used.

However Linux Packaging is how a lot of people get exposed to software.
The power of onboarding with apt-get / yum install is a big one.

I've been through the ups and downs of both approaches so many times now
in my own head, I no longer have a strong preference beyond the fact
that we do one approach today, and doing a different one is effort to
make the transition.

-Sean

It is also worth noting that according to the OpenStack User Survey [0]
56% of deployments use "Unmodifed packages from the operating system".

Granted it was a small sample size (302 responses to that question)
but it is worth keeping this in mind as we talk about moving the burden
to packagers.

0 -
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf (page
36)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

To add to this, I'd also note that I as a packager would likely stop
packaging Openstack at whatever release this goes into. While the
option to package and ship a virtualenv installed to /usr/local or /opt
exists bundling is not something that should be supported given the
issues it can have (update cadence and security issues mainly).

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Apr 18, 2016 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

So I also want to stress out that shared libraries are huge pain
during upgrades. While I'm not in favor of packages with embedded
virtualenvs (as Matt pointed out, this has a lot of issues), having
shared dependency pool pretty much means that you need to upgrade
everything that is openstack at single run, and that is prone to
errors, volatile and nearly impossible to rollback if something goes
wrong. One way to address this issue is putting services in
containers, but that is not an solution to problem at hand (56% use
apt-get install as Graham says). Packagers have hard time keeping up
already, if we add fairly complex logic to this (virtualenvs) we will
probably end up with cross-compatibility hell of people not keeping up
with changes.

That being said, in my opinion, this percentage is this high because
that's exactly what we suggest in install docs, once we came out with
a solution we should fix it there as well.

On 18 April 2016 at 10:23, Matthew Thode prometheanfire@gentoo.org wrote:
On 04/18/2016 08:24 AM, Hayes, Graham wrote:

On 18/04/2016 13:51, Sean Dague wrote:

On 04/18/2016 08:22 AM, Chris Dent wrote:

On Mon, 18 Apr 2016, Sean Dague wrote:

So if you have strong feelings and ideas, why not get them out in email
now? That will help in the framing of the conversation.

I won't be at summit and I feel pretty strongly about this topic, so
I'll throw out my comments:

I agree with the basic premise: In the big tent universe co-
installability is holding us back and is a huge cost in terms of spent
energy. In a world where service isolation is desirable and common
(whether by virtualenv, containers, different hosts, etc) targeting an
all-in-one install seems only to serve the purposes of all-in-one rpm-
or deb-based installations.

Many (most?) people won't be doing those kinds of installations. If all-in-
one installations are important to the rpm- and deb- based distributions
then they should be resolving the dependency issues local to their own
infrastructure (or realizing that it is too painful and start
containerizing or otherwise as well).

I think making these changes will help to improve and strengthen the
boundaries and contracts between services. If not technically then
at least socially, in the sense that the negotiations that people
make to get things to work are about what actually matters in their
services, not unwinding python dependencies and the like.

A lot of the basics of getting this to work are already in place in
devstack. One challenge I've run into the past is when devstack
plugin A has made an assumption about having access to a python
script provided by devstack plugin B, but it's not on $PATH or its
dependencies are not in the site-packages visible to the current
context. The solution here is to use full paths into virtenvs.

As Chris said, doing virtualenvs on the Devstack side for services is
pretty much there. The team looked at doing this last year, then stopped
due to operator feedback.

One of the things that gets a little weird (when using devstack for
development) is if you actually want to see the impact of library
changes on the environment. As you'll need to make sure you loop and
install those libraries into every venv where they are used. This
forward reference doesn't really exist. So some tooling there will be
needed.

Middleware that's pushed from one project into another (like Ceilometer
-> Swift) is also a funny edge case that I think get funnier here.

Those are mostly implementation details, that probably have work
arounds, but would need people on them.

From a strategic perspective this would basically make traditional Linux
Packaging of OpenStack a lot harder. That might be the right call,
because traditional Linux Packaging definitely suffers from the fact
that everything on a host needs to be upgraded at the same time. For
large installs of OpenStack (especially public cloud cases) traditional
packages are definitely less used.

However Linux Packaging is how a lot of people get exposed to software.
The power of onboarding with apt-get / yum install is a big one.

I've been through the ups and downs of both approaches so many times now
in my own head, I no longer have a strong preference beyond the fact
that we do one approach today, and doing a different one is effort to
make the transition.

 -Sean

It is also worth noting that according to the OpenStack User Survey [0]
56% of deployments use "Unmodifed packages from the operating system".

Granted it was a small sample size (302 responses to that question)
but it is worth keeping this in mind as we talk about moving the burden
to packagers.

0 -
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf (page
36)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

To add to this, I'd also note that I as a packager would likely stop
packaging Openstack at whatever release this goes into. While the
option to package and ship a virtualenv installed to /usr/local or /opt
exists bundling is not something that should be supported given the
issues it can have (update cadence and security issues mainly).

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2016 by Michał_Jastrzębski (9,220 points)   1 4 5
...