settingsLogin | Registersettings

[openstack-dev] [packaging] Adding packaging as an OpenStack project

0 votes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging and upstream.
- - The goal is not to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.


    • Why doing this
      ================
      It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
      that we'd be a way more effective if we worked better together, on a
      collaborative fashion using a review process like on upstream Gerrit.
      But also, we'd like to welcome anyone, and especially the operation
      folks, to contribute and give feedback. Using Gerrit is the obvious way
      to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • - What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be constructive.

  • - Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.


    • /stackforge or /openstack namespace
      =====================================
      There's been this question floating around. Should we try joining
      directly as part of OpenStack, as many suggested, or should we go
      through a first round (during one dev cycle, until Liberty is released?)
      through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/


    • Specific packaging namespace
      ==============================
      Since distribution packaging through Git implies running a lot of git
      repository (one per package), it's been discussed that we may want to
      use a specific namespace for distributions. For example,
      /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.


    • Distribution specific git repositories or specific branches
      =============================================================
      There's 2 issues if we keep the same repository names for multiple
      distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...


    • Finally
      =========
      Finally, well, please don't discuss the color of the bike-shed for too
      long. We don't really care the color as long as we have a shed to share
      the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 27, 2015 in openstack-dev by Thomas_Goirand (18,640 points)   3 10 16

96 Responses

0 votes

Great initiative, IMO. I favour going directly to openstack-, rather than stackforge-, for the migration reason that you mention.

  Original Message  
From: Thomas Goirand
Sent: Wednesday, 27 May 2015 09:17
To: OpenStack Development Mailing List (not for usage questions)
Reply To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [packaging] Adding packaging as an OpenStack project

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging and upstream.
- - The goal is not to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.


    • Why doing this
      ================
      It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
      that we'd be a way more effective if we worked better together, on a
      collaborative fashion using a review process like on upstream Gerrit.
      But also, we'd like to welcome anyone, and especially the operation
      folks, to contribute and give feedback. Using Gerrit is the obvious way
      to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • - What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be constructive.

  • - Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.


    • /stackforge or /openstack namespace
      =====================================
      There's been this question floating around. Should we try joining
      directly as part of OpenStack, as many suggested, or should we go
      through a first round (during one dev cycle, until Liberty is released?)
      through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/


    • Specific packaging namespace
      ==============================
      Since distribution packaging through Git implies running a lot of git
      repository (one per package), it's been discussed that we may want to
      use a specific namespace for distributions. For example,
      /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.


    • Distribution specific git repositories or specific branches
      =============================================================
      There's 2 issues if we keep the same repository names for multiple
      distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...


    • Finally
      =========
      Finally, well, please don't discuss the color of the bike-shed for too
      long. We don't really care the color as long as we have a shed to share
      the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 27, 2015 by Neil_Jerram (8,580 points)   1 4 11
0 votes

Many thanks to Thomas and the other packagers for a great discussion at
the summit and this fast follow-up, explained well. Looking forward to
seeing what can be achieved!

On 27/05/15 16:14, Thomas Goirand wrote:
Hi all,

tl;dr:
- We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- The intention is to better share the workload, and improve the overall
QA for packaging and upstream.
- The goal is not to publish packages upstream
- There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.

  • Why doing this
    ================
    It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
    that we'd be a way more effective if we worked better together, on a
    collaborative fashion using a review process like on upstream Gerrit.
    But also, we'd like to welcome anyone, and especially the operation
    folks, to contribute and give feedback. Using Gerrit is the obvious way
    to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be constructive.

  • Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.

  • /stackforge or /openstack namespace
    =====================================
    There's been this question floating around. Should we try joining
    directly as part of OpenStack, as many suggested, or should we go
    through a first round (during one dev cycle, until Liberty is released?)
    through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/

  • Specific packaging namespace
    ==============================
    Since distribution packaging through Git implies running a lot of git
    repository (one per package), it's been discussed that we may want to
    use a specific namespace for distributions. For example,
    /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.

  • Distribution specific git repositories or specific branches
    =============================================================
    There's 2 issues if we keep the same repository names for multiple
    distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...

  • Finally
    =========
    Finally, well, please don't discuss the color of the bike-shed for too
    long. We don't really care the color as long as we have a shed to share
    the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 27, 2015 by Tom_Fifield (13,880 points)   2 3 4
0 votes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/27/2015 10:14 AM, Thomas Goirand wrote:
Hi all,

tl;dr: - We'd like to push distribution packaging of OpenStack on
upstream gerrit with reviews. - The intention is to better share
the workload, and improve the overall QA for packaging and
upstream. - The goal is not to publish packages upstream -
There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get
started. - There's an ongoing discussion about using a distribution
specific namespace, my own opinion here is that using
/openstack-pkg-{deb,rpm} or /stackforge-pkg-{deb,rpm} would be the
most convenient because of a number of technical reasons like the
amount of Git repository. - Finally, let's not discuss for too long
and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are
just given facts. I'm sure the reader is smart enough to guess
which is what, and we welcome anyone involved in the project to
voice an opinion if he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and
Debian people gathered and collectively expressed the will to
maintain packaging artifacts within upstream OpenStack Gerrit
infrastructure. We haven't decided all the details of the
implementation, but spent the Friday morning together with members
of the infra team (hi Paul!) trying to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the
dev list, in order to let groups who were not present at the
summit. Yes, there was a consensus among distributions that this
should happen, but still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're
welcome to join the effort.

  • Why doing this ================ It's been clear to both
    Canonical/Ubuntu teams, and Debian (ie: myself) that we'd be a way
    more effective if we worked better together, on a collaborative
    fashion using a review process like on upstream Gerrit. But also,
    we'd like to welcome anyone, and especially the operation folks, to
    contribute and give feedback. Using Gerrit is the obvious way to
    give everyone a say on what we're implementing.

I was impressed to know that Ubuntu and Debian do not share packaging.
You guys obviously should do it! A fork is always better than a new
wheel. Kudos for the initiative.

As OpenStack is welcoming every day more and more projects, it's
making even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand
not only BZR, but also Git.

We'd start by merging all of our packages that aren't core
packages (like all the non-OpenStack maintained dependencies, then
the Oslo libs, then the clients). Then we'll see how we can try
merging core packages.

Another reason is that we believe working with the infra of
OpenStack upstream will improve the overall quality of the
packages. We want to be able to run a set of tests at build time,
which we already do on each distribution, but now we want this on
every proposed patch. Later on, when we have everything implemented
and working, we may explore doing a package based CI on every
upstream patch (though, we're far from doing this, so let's not
discuss this right now please, this is a very long term goal only,
and we will have a huge improvement already before this is
implemented).

So Delorean (aka RDO master, but now also RDO/kilo) already maintains
its packages on github 1 and has a gerrit review in place (f.e. see
2) and even CI (f.e. see 3). It would be so great to see those
packages moved under the official tent and have CI running inside our
own infra.

I even thought that the idea was to start on github but then move into
stackforge/, so your initiative would be a good trigger for RDO
magicians to start working on moving under the tent.

I hope that other distributions can reuse all that is achieved by RDO
project in this regard so that tooling and workflow is shared.

For me personally, moving Delorean packages to official infra would be
a time saver since I wouldn't be forced into using different gerrit
and CI installations and other dev scripts around git to contribute to
upstream and packaging.

er
[...skip...]

  • Let's keep efficiency in mind ===============================
    Over the last few years, I've been able to maintain all of
    OpenStack in Debian with little to no external contribution. Let's
    hope that the Gerrit workflow will not slow down too much the
    packaging work, even if there's an unavoidable overhead. Hopefully,
    we can implement some liberal ACL policies for the core reviewers
    so that the Gerrit workflow don't slow down anyone too much. For
    example we may be able to create new repositories very fast, and it
    may be possible to self-approve some of the most trivial patches
    (for things like typo in a package description, adding new debconf
    translations, and such obvious fixes, we shouldn't waste our
    time).

I can talk here for RPM packages (RDO, Delorean). For neutron packages
that I care about, we have multiple engineers that also work on
neutron in upstream that also care about the packaging, providing
reviews and patches for the latter. I guess other teams may be
understaffed, as Debian one, but it all depends on the distro and the
package. I think current distro maintainers should decide what
governance model applies to their situation better.

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever)
and a too restrictive fully protected gerrit workflow that may slow
down everyone too much. Currently, there's a small amount of people
involved in the packaging. While we can expect a lot of operators
will be interested to work on core packages like Nova, Neutron and
so on, I am at the same time expecting that nobody will care about
a given indirect python module dependency (and I may still continue
to be the only one working on these...). We really don't want to be
stuck in situations with nobody reviewing these.

  • /stackforge or /openstack namespace
    ===================================== There's been this question
    floating around. Should we try joining directly as part of
    OpenStack, as many suggested, or should we go through a first round
    (during one dev cycle, until Liberty is released?) through
    stackforge.

I have to admit that I'm not strongly opinionated about this
myself. Sure, being an official OpenStack big-tent project would be
nice, and it would feel worm, but also there's the issue that we
don't have any commits within the project yet, and therefore
finding who's core and who's the PTL could be an issue if we want
to follow the official guidelines.

However, without even discussing the topic between us, it's obvious
that people like James Page (plus others from his team) and myself
would be core approvers for the Debian packaging. Fedora guys can
decide for themselves (I must write names I have in mind for the
RPM based-OS side of things: M. Runge, Haikel Gemmar and Alan
Pevec, for example. Please add the missing ones here...).

Another thing is that going for stackforge now means that one day,
we'll have to suffer from a migration to the openstack namespace,
which may be 1/ a lot of work and 2/ disruptive. Avoiding such a
migration would be good.

All together, the packaging team will happily accept whatever is
decided by the TC. What counts for us is that a decision is taken
quickly, so that we can start implementing it and share our
repositories. So, dear TC, please let us know ASAP after your next
meeting.

I would vote for the stable namespace, whatever it is, and wouldn't
bother staying in stackforge/ indefinitely. If people will eventually
try to get into openstack/ for whatever reason, then let's just start
with that namespace to avoid the switch hassle later.

Anyhow, please feel free to discuss the issue within the
governance patch which I started:
https://review.openstack.org/#/c/185187/

  • Specific packaging namespace ============================== Since
    distribution packaging through Git implies running a lot of git
    repository (one per package), it's been discussed that we may want
    to use a specific namespace for distributions. For example,
    /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the
distributions, we're talking about more than 200 packages. 235 so
far on the OpenStack packaging group in Debian 1, and it's
increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we
already have a lot of projects inside /stackforge or /openstack,
and that searching through them isn't easy. Now, consider that
we'll have duplicates for each and every project. Also, we don't
have a complete match between OpenStack package names. For example,
in Debian/Ubuntu, we have "python-" as prefix for each and every
python libraries (including for example all oslo packages, but not
only). Plus we'd be forced to use a prefix, for example "pkg-nova",
to avoid collision. However, it'd be nice to keep the same
repository names as per the name of the source package in the
distributions, otherwise we'd have to fix our tooling which
obviously will become slightly more complex.

I also think that a separate namespace would be better for indexing
and searching reasons.

  • Distribution specific git repositories or specific branches
    =============================================================
    There's 2 issues if we keep the same repository names for multiple
    distribution.

First, we'd like to have a different set of core reviewers for a
given distribution family. This implies that, if we use the same
repository for RPM and DEB, then we'd have to set ACLs on
per-branch basis, which may be harder to maintain (compared to
giving access to the full set of Git repositories for a given
distro family). Yes, we could give access to both for core
reviewers, and trust them to behave. This trust model has been
proved as successful within the Debian community, but still, adding
and removing ACLs for new core reviewers would be harder to
maintain.

Then, as written above, it is desirable to have a matching pair
between distribution source package names and the name of the
underlying git repository (so that for building X, you git clone X,
not prefix-X). However, distributions don't have the same names for
packages. For example, RPM supports upper case, while dpkg systems
don't. Also, Red Hat uses a prefix "openstack-" for all core
projects (like "openstack-nova"), while Debian based distribution
don't, and so on...

I wouldn't mind to store neutron rpm packages under 'neutron' repo as
long as its artifacts are named with openstack- prefix. If that would
help to get common space to collaborate cross-distro, I'm fine with that
.

  • Finally ========= Finally, well, please don't discuss the color
    of the bike-shed for too long. We don't really care the color as
    long as we have a shed to share the hosting of our packages Git.

I have all kinds of color suggestions, but I will save them for later.

Ihar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJVZcdrAAoJEC5aWaUY1u57+8UH/2x5n1LSWpBVWn1twfhHxJfb
ejHNrL0YjaZjsQyH2t4sNfqkvI5BzY1DysOjEXJIpnSMgVua3Q8o097rZbRVKcGD
5SnU/SDI9dXy8jyDUPRH89Gnxq30x3wlC5SUr1PBglCq5RibqDDwl+nvdFyQodCc
mtUZ91hNG2QZ/Ittex954x1/4qGlp650YdS236HPrt6psNyYOZf1nyRx6z7CYRBc
QJX5JNs42aOvKVHw92dCKmYOXmqf4q3RFvGr+XuF1VvIdJZF+B+weKQ6lYSZfXB4
QyuRstWaULXAQuuYlFxaxN2d9cRw25bySmZSEL0s99LTo66fOv4E4oc9q47P/OQ=
=JMux
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 27, 2015 by Ihar_Hrachyshka (35,300 points)   3 9 16
0 votes

On 27/05/15 09:14, Thomas Goirand wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging and upstream.
- - The goal is not to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.


    • Why doing this
      ================
      It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
      that we'd be a way more effective if we worked better together, on a
      collaborative fashion using a review process like on upstream Gerrit.
      But also, we'd like to welcome anyone, and especially the operation
      folks, to contribute and give feedback. Using Gerrit is the obvious way
      to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • - What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be constructive.

  • - Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.


    • /stackforge or /openstack namespace
      =====================================
      There's been this question floating around. Should we try joining
      directly as part of OpenStack, as many suggested, or should we go
      through a first round (during one dev cycle, until Liberty is released?)
      through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/


    • Specific packaging namespace
      ==============================
      Since distribution packaging through Git implies running a lot of git
      repository (one per package), it's been discussed that we may want to
      use a specific namespace for distributions. For example,
      /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.


    • Distribution specific git repositories or specific branches
      =============================================================
      There's 2 issues if we keep the same repository names for multiple
      distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...


    • Finally
      =========
      Finally, well, please don't discuss the color of the bike-shed for too
      long. We don't really care the color as long as we have a shed to share
      the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

This all sounds great thanks for kicking it off

Ihar gave some good points of how we are doing most of this in RDO, I'll
summarize this a bit here and add a proposal of how I think we can make
some more progress towards your goal

As part of the RDO project we have been doing pretty much most of what
you described for the last 6 months, because of the number of git
repositories involved we decided to trial it on github[1] using
gerrithub[2] for peer review. Effectively we now have a yum repository
representing the collection of openstack projects per upstream commit
for both Fedora and Centos[3][4]

After some initial teething problems I think we have settled on a
process that appears to be working.

The tool which we are using to tie the flow together is called
delorean[5], essentially it monitors git repositories and triggers a
build when a commit occurs on either the upstream project or the
packaging itself, it shouldn't be too hard to expand it to build deb's
in what ever way is most appropriate. The main thing I think that is
important here is that we all agree on a common flow. We will also work
on adding support so it can build packages based on ZUUL REFs so we can
support CI triggered by upstream gerrit.

I propose we do the following,

  1. rename all of the packaging repositories in[1] to prefix them with
    something like rdorpm-

  2. Approach infra to explore the possibility of having a new namespace
    for this, I think the number of repositories would justify it (depending
    on what dependencies are needed it could be well over 100 per distro),
    if they agree then we could give infra control of the
    "openstack-packages" github organization, or alternatively create a new one.

  3. for rpm trunk builds we can point our tools at the new location and
    continue as we have been. This would have to be changed in rdoinfo[6]
    and is currently RDO specific so so we would need some work here if
    adding support to delorean for other distros.

  4. For deb packages you can create new repositories along side the
    rdorpm-* repositories

    From a packaging point of view I think the collaboration may stop there
    but from the point of view of flow around the packaging we can (and
    should) continue to collaborate e.g.

    1. Add deb support to delorean, I know of at least one person who has
      already explored this (steve cc'd), if delorean is too far off the path
      of what we want to achieve and there is a better tool then I'm open to
      change.

    2. Explore what ci jobs could be run against the packaging

The trunk repositories have proven very valuable to us over the last few
months, I hope we can make this work in a way that will be valuable to a
wider audience. How would this sound to you?

thanks,
Derek.

[1] - https://github.com/openstack-packages
[2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z
[3] - http://trunk.rdoproject.org/f21/report.html
[4] - http://trunk.rdoproject.org/centos/report.html
[5] - https://github.com/openstack-packages/delorean
[6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 27, 2015 by Derek_Higgins (5,340 points)   1 3 3
0 votes

Derek,

Thanks for what you wrote.

On 05/27/2015 11:26 PM, Derek Higgins wrote:

  1. For deb packages you can create new repositories along side the
    rdorpm-* repositories

My intention is to use deb-* as prefix, if Canonical team agrees.

  1. Add deb support to delorean, I know of at least one person who has
    already explored this (steve cc'd), if delorean is too far off the path
    of what we want to achieve and there is a better tool then I'm open to
    change.

I don't know delorean at all, but what should be kept in mind is that,
for Debian and Ubuntu, we must use sbuild, which is what is used on
the buildd networks.

I also started working on openstack-pkg-tools to provide such sbuild
based build env, so I'm not sure if we need to switch to Delorean. Could
you point me to some documentation about it, so I can see by myself what
Delorean is about?

Cheers,

Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by Thomas_Goirand (18,640 points)   3 10 16
0 votes

Hi Thomas,

Delorean is a tool to build rpm packages from master branches (maybe any
branch?) of OpenStack projects.

Check out here:
https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide

Regards,

On Thu, 28 May 2015 10:40, Thomas Goirand wrote:
Derek,

Thanks for what you wrote.

On 05/27/2015 11:26 PM, Derek Higgins wrote:

  1. For deb packages you can create new repositories along side the
    rdorpm-* repositories

My intention is to use deb-* as prefix, if Canonical team agrees.

  1. Add deb support to delorean, I know of at least one person who has
    already explored this (steve cc'd), if delorean is too far off the path
    of what we want to achieve and there is a better tool then I'm open to
    change.

I don't know delorean at all, but what should be kept in mind is that,
for Debian and Ubuntu, we must use sbuild, which is what is used on
the buildd networks.

I also started working on openstack-pkg-tools to provide such sbuild
based build env, so I'm not sure if we need to switch to Delorean. Could
you point me to some documentation about it, so I can see by myself what
Delorean is about?

Cheers,

Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Jaume Devesa
Software Engineer at Midokura


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by Jaume_Devesa (1,820 points)   1 3
0 votes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/28/2015 01:07 PM, Jaume Devesa wrote:
Hi Thomas,

Delorean is a tool to build rpm packages from master branches
(maybe any branch?) of OpenStack projects.

It's now also used for stable/kilo. I suspect all supported branches
starting from Kilo will be consumed by Delorean.

Ihar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJVZwRLAAoJEC5aWaUY1u57jEcH+gKPNa0RNfMmSUrJTI3fD+SK
7vHV1qRUww3VNzJbSF0Tdr8Zp9KsIIarXgZMyCPM4i/0Syy1Gr5PVCVnRX8TjK/V
g5NwQ69KXAiEbSCrbUJVZx4fkC4SW3E9h/UPkkaCd2IYniTv7/KeoKRSX1lHHxrl
eQSBxn4gyl2r3F7Ilq5uPb1fYSOXonlGwIFEFuyS/FjXppZ1SK0V+A3DIUavt8Tz
27S8HqC6oxwbac/iABgSYF1f0vp//W0NJSosgTV2PUrlycBLTX7o4uAe/0dEmc7K
z+aTIaa3rrb44htzrGYIF4T3JHencASoPpDq/yRa7mhSX7EHjRENV7154pJJ75U=
=IrKJ
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by Ihar_Hrachyshka (35,300 points)   3 9 16
0 votes

On 28/05/15 12:07, Jaume Devesa wrote:
Hi Thomas,

Delorean is a tool to build rpm packages from master branches (maybe any
branch?) of OpenStack projects.

Check out here:
https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide

Following those instructions you'll notice that the rpm's are being
built using rpmbuild inside a docker container, if expanding to add dep
support this is where we could plug in sbuild.

Regards,

On Thu, 28 May 2015 10:40, Thomas Goirand wrote:

Derek,

Thanks for what you wrote.

On 05/27/2015 11:26 PM, Derek Higgins wrote:

  1. For deb packages you can create new repositories along side the
    rdorpm-* repositories

My intention is to use deb-* as prefix, if Canonical team agrees.

  1. Add deb support to delorean, I know of at least one person who has
    already explored this (steve cc'd), if delorean is too far off the path
    of what we want to achieve and there is a better tool then I'm open to
    change.

I don't know delorean at all, but what should be kept in mind is that,
for Debian and Ubuntu, we must use sbuild, which is what is used on
the buildd networks.

I also started working on openstack-pkg-tools to provide such sbuild
based build env, so I'm not sure if we need to switch to Delorean. Could
you point me to some documentation about it, so I can see by myself what
Delorean is about?

Cheers,

Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by Derek_Higgins (5,340 points)   1 3 3
0 votes

On 05/27/2015 05:26 PM, Derek Higgins wrote:
On 27/05/15 09:14, Thomas Goirand wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the
overall
QA for packaging and upstream.
- - The goal is not to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.


    • Why doing this
      ================
      It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
      that we'd be a way more effective if we worked better together, on a
      collaborative fashion using a review process like on upstream Gerrit.
      But also, we'd like to welcome anyone, and especially the operation
      folks, to contribute and give feedback. Using Gerrit is the obvious way
      to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • - What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be
    constructive.

  • - Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.


    • /stackforge or /openstack namespace
      =====================================
      There's been this question floating around. Should we try joining
      directly as part of OpenStack, as many suggested, or should we go
      through a first round (during one dev cycle, until Liberty is released?)
      through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be
good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/


    • Specific packaging namespace
      ==============================
      Since distribution packaging through Git implies running a lot of git
      repository (one per package), it's been discussed that we may want to
      use a specific namespace for distributions. For example,
      /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.


    • Distribution specific git repositories or specific branches
      =============================================================
      There's 2 issues if we keep the same repository names for multiple
      distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to
maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...


    • Finally
      =========
      Finally, well, please don't discuss the color of the bike-shed for too
      long. We don't really care the color as long as we have a shed to share
      the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

This all sounds great thanks for kicking it off

Ihar gave some good points of how we are doing most of this in RDO, I'll
summarize this a bit here and add a proposal of how I think we can make
some more progress towards your goal

As part of the RDO project we have been doing pretty much most of what
you described for the last 6 months, because of the number of git
repositories involved we decided to trial it on github[1] using
gerrithub[2] for peer review. Effectively we now have a yum repository
representing the collection of openstack projects per upstream commit
for both Fedora and Centos[3][4]

After some initial teething problems I think we have settled on a
process that appears to be working.

The tool which we are using to tie the flow together is called
delorean[5], essentially it monitors git repositories and triggers a
build when a commit occurs on either the upstream project or the
packaging itself, it shouldn't be too hard to expand it to build deb's
in what ever way is most appropriate. The main thing I think that is
important here is that we all agree on a common flow. We will also work
on adding support so it can build packages based on ZUUL REFs so we can
support CI triggered by upstream gerrit.

I propose we do the following,

  1. rename all of the packaging repositories in[1] to prefix them with
    something like rdorpm-

Not sure I'm a fan of rdorpm, seems too specific to RDO and would not
foster other people using the git repo for packaging. Personally, I
simple say rpm- prefix, allowing for branches to be used for distro
specific changes.

  1. Approach infra to explore the possibility of having a new namespace
    for this, I think the number of repositories would justify it (depending
    on what dependencies are needed it could be well over 100 per distro),
    if they agree then we could give infra control of the
    "openstack-packages" github organization, or alternatively create a new
    one.

  2. for rpm trunk builds we can point our tools at the new location and
    continue as we have been. This would have to be changed in rdoinfo[6]
    and is currently RDO specific so so we would need some work here if
    adding support to delorean for other distros.

+1

  1. For deb packages you can create new repositories along side the
    rdorpm-* repositories

    From a packaging point of view I think the collaboration may stop there
    but from the point of view of flow around the packaging we can (and
    should) continue to collaborate e.g.

    1. Add deb support to delorean, I know of at least one person who has
      already explored this (steve cc'd), if delorean is too far off the path
      of what we want to achieve and there is a better tool then I'm open to
      change.

Personally, I'm a fan of mock. Is there plan to add support for it?
Also, currently containers are not being used in -infra. Not saying it
is a show stopper, but could see some initial planning that is required
for it.

  1. Explore what ci jobs could be run against the packaging

The trunk repositories have proven very valuable to us over the last few
months, I hope we can make this work in a way that will be valuable to a
wider audience. How would this sound to you?

thanks,
Derek.

[1] - https://github.com/openstack-packages
[2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z
[3] - http://trunk.rdoproject.org/f21/report.html
[4] - http://trunk.rdoproject.org/centos/report.html
[5] - https://github.com/openstack-packages/delorean
[6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by pabelanger_at_redhat (6,560 points)   1 1 2
0 votes

2015-05-27 23:26 GMT+02:00 Derek Higgins derekh@redhat.com:

On 27/05/15 09:14, Thomas Goirand wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging and upstream.
- - The goal is not to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.


    • Why doing this
      ================
      It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
      that we'd be a way more effective if we worked better together, on a
      collaborative fashion using a review process like on upstream Gerrit.
      But also, we'd like to welcome anyone, and especially the operation
      folks, to contribute and give feedback. Using Gerrit is the obvious way
      to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already before
this is implemented).

  • - What it will not be

    We do not have the intention (yet?) to publish the resulting packages
    built on upstream infra. Yes, we will share the same Git repositories,
    and yes, the infra will need to keep a copy of all builds (for example,
    because core packages will need oslo.db to build and run unit tests).
    But we will still upload on each distributions on separate repositories.
    So published packages by the infra isn't currently discussed. We could
    get to this topic once everything is implemented, which may be nice
    (because we'd have packages following trunk), though please, refrain to
    engage in this topic right now: having the implementation done is more
    important for the moment. Let's try to stay on tracks and be constructive.

  • - Let's keep efficiency in mind

    Over the last few years, I've been able to maintain all of OpenStack in
    Debian with little to no external contribution. Let's hope that the
    Gerrit workflow will not slow down too much the packaging work, even if
    there's an unavoidable overhead. Hopefully, we can implement some
    liberal ACL policies for the core reviewers so that the Gerrit workflow
    don't slow down anyone too much. For example we may be able to create
    new repositories very fast, and it may be possible to self-approve some
    of the most trivial patches (for things like typo in a package
    description, adding new debconf translations, and such obvious fixes, we
    shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.


    • /stackforge or /openstack namespace
      =====================================
      There's been this question floating around. Should we try joining
      directly as part of OpenStack, as many suggested, or should we go
      through a first round (during one dev cycle, until Liberty is released?)
      through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be
good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/


    • Specific packaging namespace
      ==============================
      Since distribution packaging through Git implies running a lot of git
      repository (one per package), it's been discussed that we may want to
      use a specific namespace for distributions. For example,
      /stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.


    • Distribution specific git repositories or specific branches
      =============================================================
      There's 2 issues if we keep the same repository names for multiple
      distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to
maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...


    • Finally
      =========
      Finally, well, please don't discuss the color of the bike-shed for too
      long. We don't really care the color as long as we have a shed to share
      the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

This all sounds great thanks for kicking it off

Ihar gave some good points of how we are doing most of this in RDO, I'll
summarize this a bit here and add a proposal of how I think we can make some
more progress towards your goal

As part of the RDO project we have been doing pretty much most of what you
described for the last 6 months, because of the number of git repositories
involved we decided to trial it on github[1] using gerrithub[2] for peer
review. Effectively we now have a yum repository representing the collection
of openstack projects per upstream commit for both Fedora and Centos[3][4]

After some initial teething problems I think we have settled on a process
that appears to be working.

The tool which we are using to tie the flow together is called delorean[5],
essentially it monitors git repositories and triggers a build when a commit
occurs on either the upstream project or the packaging itself, it shouldn't
be too hard to expand it to build deb's in what ever way is most
appropriate. The main thing I think that is important here is that we all
agree on a common flow. We will also work on adding support so it can build
packages based on ZUUL REFs so we can support CI triggered by upstream
gerrit.

I propose we do the following,

  1. rename all of the packaging repositories in[1] to prefix them with
    something like rdorpm-

couldn't we use branches and just use rpm- prefix ?
If OpenSuse people could give their insight, I prefer sharing a common
namespace.

  1. Approach infra to explore the possibility of having a new namespace for
    this, I think the number of repositories would justify it (depending on what
    dependencies are needed it could be well over 100 per distro), if they agree
    then we could give infra control of the "openstack-packages" github
    organization, or alternatively create a new one.

+1

  1. for rpm trunk builds we can point our tools at the new location and
    continue as we have been. This would have to be changed in rdoinfo[6] and is
    currently RDO specific so so we would need some work here if adding support
    to delorean for other distros.

+1

  1. For deb packages you can create new repositories along side the rdorpm-*
    repositories

From a packaging point of view I think the collaboration may stop there but
from the point of view of flow around the packaging we can (and should)
continue to collaborate e.g.

  1. Add deb support to delorean, I know of at least one person who has
    already explored this (steve cc'd), if delorean is too far off the path of
    what we want to achieve and there is a better tool then I'm open to change.

+1

  1. Explore what ci jobs could be run against the packaging

+1

The trunk repositories have proven very valuable to us over the last few
months, I hope we can make this work in a way that will be valuable to a
wider audience. How would this sound to you?

Yes, thanks to master builds, we were able to significantly increase
robustness of RDO.
I encourage everyone to follow that path.

H.

thanks,
Derek.

[1] - https://github.com/openstack-packages
[2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z
[3] - http://trunk.rdoproject.org/f21/report.html
[4] - http://trunk.rdoproject.org/centos/report.html
[5] - https://github.com/openstack-packages/delorean
[6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml

Cheers,

Thomas Goirand (zigo)

[1]

https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 28, 2015 by Haïkel (1,800 points)   2
...