settingsLogin | Registersettings

[openstack-dev] [oslo] Can we stop global requirements update?

0 votes

Hoy,

So Gnocchi gate is all broken (agaiiiin) because it depends on "pbr" and
some new release of oslo.* depends on pbr!=2.1.0.

Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0
that got in banished by requirements Gods. It does not prevent it to be
used e.g. to install the software or get version information. But it
does break anything that is not in OpenStack because well, pip installs
the latest pbr (2.1.0) and then oslo.* is unhappy about it.

So I understand the culprit is probably pip installation scheme, and we
can blame him until we fix it. I'm also trying to push pbr 2.2.0 to
avoid the entire issue.

But for the future, could we stop updating the requirements in oslo libs
for no good reason? just because some random OpenStack project hit a bug
somewhere?

For example, I've removed requirements update on tooz¹ for more than a
year now, which did not break anything in the meantime, proving that
this process is giving more problem than solutions. Oslo libs doing that
automatic update introduce more pain for all consumers than anything (at
least not in OpenStack).

So if we care about Oslo users outside OpenStack, I beg us to stop this
crazyness. If we don't, we'll just spend time getting rid of Oslo over
the long term…

My 2c,

Cheers,

¹ Unless some API changed in a dep and we needed to raise the dep,
obviously.

--
Julien Danjou

Free Software hacker

https://julien.danjou.info


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked May 19, 2017 in openstack-dev by Julien_Danjou (20,500 points)   2 4 6

30 Responses

0 votes

Excerpts from Joshua Harlow's message of 2017-04-20 22:31:19 -0700:

Doug Hellmann wrote:

Excerpts from gordon chung's message of 2017-04-20 17:12:26 +0000:

On 20/04/17 01:32 AM, Joshua Harlow wrote:

Wasn't there also some decision made in austin (?) about how we as a
group stated something along the line of co-installability isn't as
important as it once was (and may not even be practical or what people
care about anymore anyway)?

I don't remember that, but I may not have been in the room at the
time. In the past when we've discussed that idea, we've continued
to maintain that co-installability is still needed for distributors
who have packaging constraints that require it and for use cases
like single-node deployments for POCs.

Ya, looking back I think it was:

https://etherpad.openstack.org/p/newton-global-requirements

I think that was robert that lead that session, but I might be incorrect
there.

That was me, though Robert was definitely present and vocal.

My memory of the outcome of that session was that we needed to maintain
co-installability; that we could continue to keep an eye on the
container space as an alternative; and that a new team of maintainers
would take over the requirements list (which was my secret agenda for
proposing that we stop doing it at all).

During the session in Barcelona (I previously said Austin, but
misremembered the location) we agreed that we could stop syncing,
as long as we maintained co-installability by ensuring that everyone's
requirements lists intersect with the upper-constraints.txt list. That
work has been started.

As far as I know, we have never said we could drop co-installability as
a requirement. We have wished we could, but have not said we can.

Doug

With kolla becoming more popular (tripleo I think is using it, and ...)
and the containers it creates making isolated per-application
environments it makes me wonder what of global-requirements is still
valid (as a concept) and what isn't.

We still need to review dependencies for license compatibility, to
minimize redundancy, and to ensure that we're not adding things to
the list that are not being maintained upstream. Even if we stop syncing
versions, official projects need to be those reviews, and having the
global list is a way to ensure that the reviews are done.

I do remember the days of free for all requirements (or requirements
sometimes just put/stashed in devstack vs elsewhere), which I don't
really want to go back to; but if we finally all agree that
co-installability isn't what people actually do and/or care about
(anymore?) then maybe we can re-think some things?
agree with all of ^... but i imagine to make progress on this, we'd have
to change/drop devstack usage in gate and that will take forever and a
lifetime (is that a chick flick title?) given how embedded devstack is
in everything. it seems like the solution starts with devstack.

cheers,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 21, 2017 by Doug_Hellmann (87,520 points)   3 4 12
0 votes

On Fri, Apr 21 2017, Doug Hellmann wrote:

My memory of the outcome of that session was that we needed to maintain
co-installability; that we could continue to keep an eye on the
container space as an alternative; and that a new team of maintainers
would take over the requirements list (which was my secret agenda for
proposing that we stop doing it at all).

Just to come back to the original topic, co-installability is not
impacted by my proposal of stopping the sync. On the opposite, last week
it was very hard to install both Gnocchi and Oslo because of Oslo…

So it would improve co-installability, not remove it.

My 2c,

--
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Apr 22, 2017 by Julien_Danjou (20,500 points)   2 4 6
0 votes

On Wed, Apr 19 2017, Julien Danjou wrote:

So Gnocchi gate is all broken (agaiiiin) because it depends on "pbr" and
some new release of oslo.* depends on pbr!=2.1.0.

Same things happened today with Babel. As far as Gnocchi is concerned,
we're going to take the easiest route and remove all our oslo
dependencies over the next months for sanely maintained alternative at
this point.

Cheers,
--
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 16, 2017 by Julien_Danjou (20,500 points)   2 4 6
0 votes

On 2017-05-16 11:42, Julien Danjou wrote:
On Wed, Apr 19 2017, Julien Danjou wrote:

So Gnocchi gate is all broken (agaiiiin) because it depends on "pbr" and
some new release of oslo.* depends on pbr!=2.1.0.

Same things happened today with Babel. As far as Gnocchi is concerned,
we're going to take the easiest route and remove all our oslo
dependencies over the next months for sanely maintained alternative at
this point.

what exactly happened with Babel?

I see in global-requirements the following:
Babel>=2.3.4,!=2.4.0 # BSD

that shouldn't case a problem - or does it? Or what's the problem?

Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 16, 2017 by Andreas_Jaeger (17,140 points)   2 3 4
0 votes

On Tue, May 16 2017, Andreas Jaeger wrote:

what exactly happened with Babel?

I see in global-requirements the following:
Babel>=2.3.4,!=2.4.0 # BSD

that shouldn't case a problem - or does it? Or what's the problem?

Damn, at the moment I pressed the `Sent' button I thought "You just
complained without including much detail idiot". Sorry about that!

One of the log that fails:

http://logs.openstack.org/13/464713/2/check/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/db61bdf/console.html

Basically oslo.policy pulls oslo.i18n which pulls Babel!=2.4.0
But Babel is already pulled by os-testr which depends on >=2.3.4.
So pip does not solve that (unfortunately) and then the failure is:

2017-05-16 05:08:43.629772 | 2017-05-16 05:08:43.503 10699 ERROR gnocchi
ContextualVersionConflict: (Babel 2.4.0
(/home/jenkins/workspace/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/upgrade/lib/python2.7/site-packages),
Requirement.parse('Babel!=2.4.0,>=2.3.4'), set(['oslo.i18n']))

I'm pretty sure Babel should not even be in the requirements list of
oslo.i18n since it's not a runtime dependency AFAIU.

--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 16, 2017 by Julien_Danjou (20,500 points)   2 4 6
0 votes

On 2017-05-16 12:10, Julien Danjou wrote:
On Tue, May 16 2017, Andreas Jaeger wrote:

what exactly happened with Babel?

I see in global-requirements the following:
Babel>=2.3.4,!=2.4.0 # BSD

that shouldn't case a problem - or does it? Or what's the problem?

Damn, at the moment I pressed the `Sent' button I thought "You just
complained without including much detail idiot". Sorry about that!

no worries.

One of the log that fails:

http://logs.openstack.org/13/464713/2/check/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/db61bdf/console.html

Basically oslo.policy pulls oslo.i18n which pulls Babel!=2.4.0
But Babel is already pulled by os-testr which depends on >=2.3.4.

and os-testr is not importing global-requirements:
https://review.openstack.org/#/c/454511/

So pip does not solve that (unfortunately) and then the failure is:

2017-05-16 05:08:43.629772 | 2017-05-16 05:08:43.503 10699 ERROR gnocchi
ContextualVersionConflict: (Babel 2.4.0
(/home/jenkins/workspace/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/upgrade/lib/python2.7/site-packages),
Requirement.parse('Babel!=2.4.0,>=2.3.4'), set(['oslo.i18n']))

I'm pretty sure Babel should not even be in the requirements list of
oslo.i18n since it's not a runtime dependency AFAIU.

It is needed to generate the translations, but can't we move it for
oslo-i18n into test-requirements?

But os-testr does not need Babel at all - let's remove it,
https://review.openstack.org/465023

Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 16, 2017 by Andreas_Jaeger (17,140 points)   2 3 4
0 votes

On Tue, May 16 2017, Andreas Jaeger wrote:

It is needed to generate the translations, but can't we move it for
oslo-i18n into test-requirements?

I've pushed this and it seems to work, pretty sure it's safe.

https://review.openstack.org/#/c/465014/

If we can merge this today and then release quickly after that'd be a
great help -_-

But os-testr does not need Babel at all - let's remove it,
https://review.openstack.org/465023

Arf, sure!
I can only +1 though :(

--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 16, 2017 by Julien_Danjou (20,500 points)   2 4 6
0 votes

On 05/16/2017 05:42 AM, Julien Danjou wrote:
On Wed, Apr 19 2017, Julien Danjou wrote:

So Gnocchi gate is all broken (agaiiiin) because it depends on "pbr" and
some new release of oslo.* depends on pbr!=2.1.0.

Same things happened today with Babel. As far as Gnocchi is concerned,
we're going to take the easiest route and remove all our oslo
dependencies over the next months for sanely maintained alternative at
this point.

I'm not understanding this? do you mean this?

diff --git a/gnocchi/indexer/sqlalchemy.py b/gnocchi/indexer/sqlalchemy.py
index 3497b52..0ae99fd 100644
--- a/gnocchi/indexer/sqlalchemy.py
+++ b/gnocchi/indexer/sqlalchemy.py
@@ -22,11 +22,7 @@ import uuid

from alembic import migration
from alembic import operations
-import oslodb.api
-from oslo
db import exception
-from oslodb.sqlalchemy import enginefacade
-from oslo
db.sqlalchemy import utils as oslodbutils
-from oslo_log import log
+from ??? import ???
try:
import psycopg2
except ImportError:

Cheers,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 18, 2017 by Mike_Bayer (15,260 points)   1 5 6
0 votes

On Thu, May 18 2017, Mike Bayer wrote:

I'm not understanding this? do you mean this?

In the long run, yes. Unfortunately, we're not happy with the way Oslo
libraries are managed and too OpenStack centric. I've tried for the last
couple of years to move things on, but it's barely possible to deprecate
anything and contribute, so I feel it's safer to start fresh and better
alternative. Cotyledon by Mehdi is a good example of what can be
achieved.

Though to comment on your example, oslo.db is probably the most useful
Oslo library that Gnocchi depends on and that won't go away in a snap.
:-(

--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 18, 2017 by Julien_Danjou (20,500 points)   2 4 6
0 votes

On 05/18/2017 02:37 PM, Julien Danjou wrote:
On Thu, May 18 2017, Mike Bayer wrote:

I'm not understanding this? do you mean this?

In the long run, yes. Unfortunately, we're not happy with the way Oslo
libraries are managed and too OpenStack centric. I've tried for the last
couple of years to move things on, but it's barely possible to deprecate
anything and contribute, so I feel it's safer to start fresh and better
alternative. Cotyledon by Mehdi is a good example of what can be
achieved.

here's cotyledon:

https://cotyledon.readthedocs.io/en/latest/

replaces oslo.service with a multiprocessing approach that doesn't use
eventlet. great! any openstack service that rides on oslo.service
would like to be able to transparently switch from eventlet to
multiprocessing the same way they can more or less switch to mod_wsgi at
the moment. IMO this should be part of oslo.service itself. Docs
state: "oslo.service being impossible to fix and bringing an heavy
dependency on eventlet, " is there a discussion thread on that?

I'm finding it hard to believe that only a few years ago, everyone saw
the wisdom of not re-implementing everything in their own projects and
using a common layer like oslo, and already that whole situation is
becoming forgotten - not just for consistency, but also when a bug is
found, if fixed in oslo it gets fixed for everyone.

An increase in the scope of oslo is essential to dealing with the issue
of "complexity" in openstack. The state of openstack as dozens of
individual software projects each with their own idiosyncratic quirks,
CLIs, process and deployment models, and everything else that is visible
to operators is ground zero for perceived operator complexity.

Though to comment on your example, oslo.db is probably the most useful
Oslo library that Gnocchi depends on and that won't go away in a snap.
:-(


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 18, 2017 by Mike_Bayer (15,260 points)   1 5 6
...