settingsLogin | Registersettings

[openstack-dev] [all][pbr] splitting our deployment vs install dependencies

0 votes

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not requirements.txt
- when there are requirements in setup.cfg, stop reading requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 12, 2015 in openstack-dev by Robert_Collins (27,200 points)   4 7 13
retagged Apr 14, 2015 by admin

31 Responses

0 votes

On 04/12/2015 06:43 PM, Robert Collins wrote:
Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not requirements.txt
- when there are requirements in setup.cfg, stop reading requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 12, 2015 by Monty_Taylor (22,780 points)   2 5 8
0 votes

On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor mordred@inaugust.com wrote:

On 04/12/2015 06:43 PM, Robert Collins wrote:

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the
distribution.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in
https://review.openstack.org/#/c/161047/

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -

http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not
requirements.txt
- when there are requirements in setup.cfg, stop reading
requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by James_Polley (2,620 points)   2 3
0 votes

On 04/12/2015 08:01 PM, James Polley wrote:
On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor mordred@inaugust.com wrote:

On 04/12/2015 06:43 PM, Robert Collins wrote:

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the
distribution.

This is not accurate. During distro packaging activities, pbr does not
process dependencies at all. So no matter how we pin things in
OpenStack, it does not make it harder for the distros.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in
https://review.openstack.org/#/c/161047/

I disagree with this proposal. It's also not helping any users. Because
of what I said above, there is no flexibility that we lose downstream by
being strict and pedantic with our versions. So, having the "lose" and
the "strict" file just gets us two files and doubles the confusion.
Having a list of what we know the state to be is great. We should give
that to users. If they want to use something other than pip to install,
awesome - the person in charge of curating that content can test the
version interactions in their environment.

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -

http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not
requirements.txt
- when there are requirements in setup.cfg, stop reading
requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.

I disagree that anything is broken for us that is not caused by our
inability to remember that distro packaging concerns are not the same as
our concerns, and that the mechanism already exists for distro pacakgers
to do what they want. Furthermore, it is caused by an insistence that we
need to keep versions "open" for some ephemeral reason such as "upstream
might release a bug fix" Since we all know that "if it's not tested,
it's broken" - any changes to upstream software should be considered
broken until proven otherwise. History over the last 5 years has shown
this to be accurate more than the other thing.

If we pin the stable branches with hard pins of direct and indirect
dependencies, we can have our stable branch artifacts be installable.
Thats awesome. IF there is a bugfix release or a security update to a
dependent library - someone can propose it. Otherwise, the stable
release should not be moving.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Monty_Taylor (22,780 points)   2 5 8
0 votes

On 13 April 2015 at 12:01, James Polley jp@jamezpolley.com wrote:

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is that
our CI system wants a very tightly pinned requirements, as do downstream CI
systems and deployers that want to test and deploy exact known-tested
versions of things. On the other hand, downstream distributors (including OS
packagers) need to balance OpenStack's version requirements with version
requirements from all the other packages in their distribution; the tighter
the requirements we list are, the harder it is for the requirements to work
with the requirements of other packages in the distribution.

They are analogous yes.
...

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable branches:
a requirements.in with mandatory version constraints while being as loose as
otherwise possible, which is used to generate a requirements.txt which has
the "local to deployment" exact versions that are used in our CI. The
details of the proposal are in https://review.openstack.org/#/c/161047/

That proposal is still under discussion, and seems stuck between the
distro and -infra folk. if it ends up doing the transitive thing, I
think that that would make a sensible requirements.txt, yes. However
see the spec for that thread of discussion.
..

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

Monty:
So wheels are already in that space. metadata-2.0 is about bringing
that declarative stuff forward in the pipeline, from binary packages
to source packages. I'm currently using frustration based development
to bring it in at the start - for developers, in the lead-in to source
packages.

So you're concerned - but about what specifically? What goes wrong
with wheels (not wheels with C code). Whats not robust about the
pattern? The cargo package manager you referred to is entirely
declarative....

James: I don't think the binary distribution stuff is relevant to my
discussion, since I'm talking entirely about 'using-pip' use cases,
whereas dpkg and rpm packages don't use that at all.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Robert_Collins (27,200 points)   4 7 13
0 votes

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

We need to balance the inability to atomically update things - which
forces a degree of freedom on install_requires - with being able to
give someone the same install that we tested.

That is the fundamental tension that we're not handling well, nor have
I seen a proposal to tackle it so far.

I'll have to spend some time noodling on this, but one of the clear
constraints is that install_requires cannot both be:
- flexible enough to permit us to upgrade requirements across many
git based packages [because we could do coordinated releases of sdists
to approximate atomic bulk changes]
- tight enough enough to give the next person trying to run that ref
of the package the same things we installed in CI.

-> I think we need something other than install_requires

...

I disagree that anything is broken for us that is not caused by our
inability to remember that distro packaging concerns are not the same as
our concerns, and that the mechanism already exists for distro pacakgers
to do what they want. Furthermore, it is caused by an insistence that we
need to keep versions "open" for some ephemeral reason such as "upstream
might release a bug fix" Since we all know that "if it's not tested,
it's broken" - any changes to upstream software should be considered
broken until proven otherwise. History over the last 5 years has shown
this to be accurate more than the other thing.

This seems like a strong argument for really being able to reconstruct
what was in CI.

If we pin the stable branches with hard pins of direct and indirect
dependencies, we can have our stable branch artifacts be installable.
Thats awesome. IF there is a bugfix release or a security update to a
dependent library - someone can propose it. Otherwise, the stable
release should not be moving.

Can we do that in stable branches? We've still got the problem of
bumping dependencies across multiple packages.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Robert_Collins (27,200 points)   4 7 13
0 votes

On 13 April 2015 at 13:09, Robert Collins robertc@robertcollins.net wrote:
On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Robert_Collins (27,200 points)   4 7 13
0 votes

On 13/4/2015, at 3:53, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 13:09, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

That sounds like a very neat idea, this way we could look back, and backtrack
to discover which package version change breaks the system.

Miguel Angel Ajo


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Miguel_Ángel_Ajo (6,980 points)   1 4 7
0 votes

On Mon, 13 Apr 2015, Robert Collins wrote:

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

That seems like an excellent idea. It provides a known good set of
requirements within a broader window. This is good because:

  • It allows people who just needs things to work to have a correct
    base.
  • Doesn't disable one of the best things about flexible requirements:
    post-release broadly based integration testing. This is the best
    thing about open source software: random bugs help drive
    improvement. If we constrain things too much we are not a healthy
    and contributing participant in the larger ecosystem that is
    sometimes called "using stuff" but is also "global integration
    testing".

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Chris_Dent (7,940 points)   1 4 7
0 votes

Robert Collins wrote:
On 13 April 2015 at 13:09, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

I totally agree that we need to stop trying to provide two different
sets of dependency information (known good deps, known bad deps) using
the same dataset.

If I understand you correctly, today we provide a requirements.txt and
generate an installrequires from it, and in the new world order we
would provide a install
requires with "known-bad" info in it and
generate a "known-good" requirements.txt (during CI) from it.

Questions:

How would global-requirements evolve in that picture ? Would we have
some "global-install-requires" thing to replace it ?

Distro packagers today rely on requirements.txt (and global
-requirements) to determine what version of libraries they need to
package. Would they just rely on install_requires instead ? Where is
that information provided ? setup.cfg ?

How does this proposal affect stable branches ? In order to keep the
breakage there under control, we now have stable branches for all the
OpenStack libraries and cap accordingly[1]. We planned to cap all other
libraries to "the version that was there when the stable branch was
cut". Where would we do those cappings in the new world order ? In
install_requires ? Or should we not do that anymore ?

[1]
http://specs.openstack.org/openstack/openstack-specs/specs/library-stable-branches.html

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Thierry_Carrez (57,480 points)   3 8 15
0 votes

On 13 April 2015 at 22:04, Thierry Carrez thierry@openstack.org wrote:

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

I totally agree that we need to stop trying to provide two different
sets of dependency information (known good deps, known bad deps) using
the same dataset.

If I understand you correctly, today we provide a requirements.txt and
generate an installrequires from it, and in the new world order we
would provide a install
requires with "known-bad" info in it and
generate a "known-good" requirements.txt (during CI) from it.

Yes, with two clarifying points: the known-good has to live in a
different repo from the project, because we only discover that during
CI, after the commits have been made. Secondly, the install_requires
will be delivered via setup.cfg in the project tree.

Questions:

How would global-requirements evolve in that picture ? Would we have
some "global-install-requires" thing to replace it ?

I think global-requirements today is (by necessity) mostly known-bad,
and doesn't need to change much. It needs to learn how to reference
setup.cfg metadata as well/rather than {test-}requirements{-pyNN}.txt.
There's a separate discussion we had a few weeks back about
consolidating the non-install-requires we have into setup.cfg with
appropriate tags, which we'll want to do at the same time.

Distro packagers today rely on requirements.txt (and global
-requirements) to determine what version of libraries they need to
package. Would they just rely on install_requires instead ? Where is
that information provided ? setup.cfg ?

Yes. Project + global-requirements is a good combination. They might
want to reference the known-good exact lists as an additional data
source.

How does this proposal affect stable branches ? In order to keep the
breakage there under control, we now have stable branches for all the
OpenStack libraries and cap accordingly[1]. We planned to cap all other
libraries to "the version that was there when the stable branch was
cut". Where would we do those cappings in the new world order ? In
install_requires ? Or should we not do that anymore ?

[1]
http://specs.openstack.org/openstack/openstack-specs/specs/library-stable-branches.html

I don't think there's a hard and fast answer here. Whats proposed
there should work fine.

On the one hand, semver tells us when a backwards compat break
happens, but it doesn't tell us if that break affects user X. For
instance, the general migration pattern we expect is:
- introduce new API V=2.3.0
- migrate all our users V~=2.3
- deprecate old API V~=2.3
- gc deprecated code at some future date V>=3.0

In fact, I'd say we're hoping to never have a supported release broken
by that process... so capping just creates artificial version
conflicts which we have to resolve by issuing updates to say that
actually the new major version is still compatible with this new
release...

OTOH there will eventually be releases of our libraries that do break
prior releases of our servers/clients - and when that happens capped
requirements will actually be useful, but only to people running
unsupported releases :).

OTGH if we do deliberately break supported releases in our libraries,
then the capping process is absolutely essential.

Personally, I'd be more worried about the versions of our dependencies
that aren't coordinated with our projects, because if they aren't
capped, (and they're doing semver) we're less likely to find out the
easy way (in advance :)) about issues.

But that then swings back around to known good vs known bad. One way
of looking at that is that safe capping requires several items of
data:
- what version to use with ~= - I'm not sure that using the exact
version we got is correct. e.g. with semver, if 1.2.3 is known-good,
we should use ~=1.2 (e.g. >=1.2, ==1.*), but with date based its
harder to predict what will indicate a breaking version :). And of
course for non-semver, 1.2.3 doesn't tell us whether 1.3 will be
breaking, or even 1.2.4.
- a known good version to base our cap on

If we generated the first item and stored it somewhere, then when we
generate known-good == lists from CI, we could also generate a
known-good capped list, (e.g. transforming 1.2.3 to ~=1.2 for semver
projects). We could in principle add that to our tarball releases of
projects, even though we can't sensibly put it in git.

tl;dr - I dunno :)

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2015 by Robert_Collins (27,200 points)   4 7 13
...