settingsLogin | Registersettings

[openstack-dev] [kolla] Tags, revisions, dockerhub

0 votes

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 24, 2017 in openstack-dev by Michał_Jastrzębski (9,220 points)   1 5 5

31 Responses

0 votes

+1 to revision information in tags. If you don't have it, depending on when containers start, you could have different versions of software with/without bug fixes running on the same cluster without an easy way to know.

We've seen this with libvirt versions being 1.x in some builds and 2.x in fresher builds.

We can still keep around the unrevisioned tags too for folks that don't want to deal with the revision numbers. in which case, the unrevisioned 4.0.0 tag would map to the newest revisioned tag, say 4.0.0-2 and would have the same behavior then as is today.

Thanks,
Kevin


From: Michał Jastrzębski [inc007@gmail.com]
Sent: Wednesday, April 12, 2017 3:59 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [kolla] Tags, revisions, dockerhub

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 12, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

On Thu, Apr 13, 2017 at 10:59 AM, Michał Jastrzębski inc007@gmail.com
wrote:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

I already have a change which proposes tagging images with a pbr built
version [1]. I think if users want tags which are stable for the duration
of a major release they should switch to using the tag specified by
kolla-build.conf base_tag, which can be set to latest, ocata, pike, etc.
This would leave the version tag to at least track changes to the kolla
repo itself. Since the contents of upstream kolla images come from such
diverse sources, all I could suggest to ensure unique tags are created for
unique images is to append a datestamp to [1] (or have an extra datestamp
based tag). Bonus points for only publishing a new datestamp tag if the
contents of the image really changes.

In the RDO openstack-kolla package we now tag images with the
{Version}-{Release} of the openstack-kolla package which built it[2]. I
realise this doesn't solve the problem of the tag needing to change when
other image contents need to be updated, but I believe this can be solved
within the RDO image build pipeline by incrementing the {Release} whenever
a new image needs to be published.

[1] https://review.openstack.org/#/c/448380/
[2] https://review.rdoproject.org/r/#/c/5923/1/openstack-kolla.spec


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2017 by Steve_Baker (7,380 points)   1 3 6
0 votes

On 13.04.2017 0:59, Michał Jastrzębski wrote:
My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

+1 that this should be improved, indeed.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.

Please take a look the build flow for a neighbour project Kubernetes
[0]. I believe it addresses each of the possible versioning aspects,
like stable branch builds, master builds, infrastructure vs local builds
and so on. Kolla could reuse some build scripts and follow the similar
versioning scheme from that project and save time as well.

[0] https://github.com/kubernetes/kubernetes/tree/master/build#basic-flow

While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 13, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

Or you could use pbr's strategy, which tags based on git tags and
revision history:

https://docs.openstack.org/developer/pbr/#version

"If the currently checked out revision is not tagged, then we take the
last tagged version number and increment it to get a minimum target
version."

Best,
-jay

On 04/13/2017 04:14 AM, Bogdan Dobrelya wrote:
On 13.04.2017 0:59, Michał Jastrzębski wrote:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

+1 that this should be improved, indeed.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.

Please take a look the build flow for a neighbour project Kubernetes
[0]. I believe it addresses each of the possible versioning aspects,
like stable branch builds, master builds, infrastructure vs local builds
and so on. Kolla could reuse some build scripts and follow the similar
versioning scheme from that project and save time as well.

[0] https://github.com/kubernetes/kubernetes/tree/master/build#basic-flow

While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 14, 2017 by Jay_Pipes (59,760 points)   3 10 14
0 votes

Does reversion of deps get handled? Thats the primary concern I have. I don't
think pip/pbr handle it as deps are external to the package. Containers have
deps bundled into the actual build, so need to be accounted for somehow in the tag.

Thanks,
Kevin


From: Jay Pipes [jaypipes@gmail.com]
Sent: Friday, April 14, 2017 1:47 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] Tags, revisions, dockerhub

Or you could use pbr's strategy, which tags based on git tags and
revision history:

https://docs.openstack.org/developer/pbr/#version

"If the currently checked out revision is not tagged, then we take the
last tagged version number and increment it to get a minimum target
version."

Best,
-jay

On 04/13/2017 04:14 AM, Bogdan Dobrelya wrote:
On 13.04.2017 0:59, Michał Jastrzębski wrote:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

+1 that this should be improved, indeed.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.

Please take a look the build flow for a neighbour project Kubernetes
[0]. I believe it addresses each of the possible versioning aspects,
like stable branch builds, master builds, infrastructure vs local builds
and so on. Kolla could reuse some build scripts and follow the similar
versioning scheme from that project and save time as well.

[0] https://github.com/kubernetes/kubernetes/tree/master/build#basic-flow

While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 14, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Excerpts from Michał Jastrzębski's message of 2017-04-12 15:59:34 -0700:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal

What's in the images, kolla? Other OpenStack components? Where does the
4.0.0 come from?

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 17, 2017 by Doug_Hellmann (87,520 points)   3 4 9
0 votes

On Tue, Apr 18, 2017 at 9:57 AM, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Michał Jastrzębski's message of 2017-04-12 15:59:34 -0700:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal

What's in the images, kolla? Other OpenStack components?

Yes, each image will typically contain all software required for one
OpenStack service, including dependencies from OpenStack projects or the
base OS. Installed via some combination of git, pip, rpm, deb.

Where does the
4.0.0 come from?

Its the python version string from the kolla project itself, so ultimately
I think pbr. I'm suggesting that we switch to using the
version.release_string[1] which will tag with the longer version we use for
other dev packages.

[1]https://review.openstack.org/#/c/448380/1/kolla/common/config.py


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 17, 2017 by Steve_Baker (7,380 points)   1 3 6
0 votes

I think we have two topics and improvements here

  1. images in https://hub.docker.com/r/kolla/
  2. tag in end-user env.

images in hub.docker.com

we are building kolla tag image and push them into hub.docker.com. After
this,
we do nothing for these images.

The issue is

  1. any security update is not included in these images.
    solution: I do not think use 4.0.0-1 4.0.0-2 in hub.docker.com is a good
    idea.
    if so, we need mark what 4.0.0-1 container and what's the difference
    with 4.0.0-2.
    This will make another chaos.
    And any prod env shouldn't depend on hub.docker.com's images, which is
    vulnerable
    to attack and is mutable.

  2. branch images are not pushed.
    solution: we can add a job to push branch images into hub.docker.com
    like inc0
    said. For example:
    centos-source-nova-api:4.0.0
    centos-source-nova-api:ocata
    centos-source-nova-api:pike
    centos-source-nova-api:master
    But branch tag images is not stable ( even its name is stable/ocata ),
    users are
    not recommended to use these images

images in end-user env

I recommended end user should build its own image rather then use
hub.docker.com directly.
in my env, I build images with following tag rule.

when using 4.0.0 to build multi time, i use different tag name. For example
1st: 4.0.0.1
2nd: 4.0.0.2
3rd: 4.0.0.3
...

The advantage in this way is: keep each tag as immutable ( never override )

On Tue, Apr 18, 2017 at 6:46 AM, Steve Baker sbaker@redhat.com wrote:

On Tue, Apr 18, 2017 at 9:57 AM, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Michał Jastrzębski's message of 2017-04-12 15:59:34 -0700:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal

What's in the images, kolla? Other OpenStack components?

Yes, each image will typically contain all software required for one
OpenStack service, including dependencies from OpenStack projects or the
base OS. Installed via some combination of git, pip, rpm, deb.

Where does the
4.0.0 come from?

Its the python version string from the kolla project itself, so ultimately
I think pbr. I'm suggesting that we switch to using the
version.release_string[1] which will tag with the longer version we use for
other dev packages.

[1]https://review.openstack.org/#/c/448380/1/kolla/common/config.py


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2017 by Lei_Zhang (6,360 points)   1 2 5
0 votes

So, while I agree that everyone should use images built locally, I
also see what is value of downloading dockerhub-hosted images (besides
speed). Images at dockerhub passed our gates and are CI tested. Which
means quality of these images is ensured by our CI. Not everyone can
afford to have CI/CD system in their infra, so for small/medium
installation this actually might be more stable than local builds.

Given that both local and dockerhub hosted images have valid
production use case, I'd like that we keep our tagging mechanism same
for both.
That makes revisions per se impossible to handle (4.0.0-3 on dockerhub
doesn't mean 4.0.0-3 locally). Also how often we push 4.0.0-x? Daily
for quickest update on security patches (preferably for me), that
would mean that our dockerhub registry would grow extremely fast if
we'd like to retain every revision. One idea would be to put a little
of this weight on users (sorry!). We could upload daily :ocata images
and delete old ones. I think not many people does daily openstack
upgrades (cool use case tho), that means they will have some stale
:ocata image locally. We can just put an expectation to backup your
images locally, ones that you actually use. Just tar.gz
/var/lib/docker/volumes/registry/_data and save it somewhere safe.
Then you can always come back to it.

Bottom line, my suggestion for now is to have schema like that:

image-name:4.0.0 -> corresponding docker tag and image built (pref
automatically) close to tagging date
image-name:ocata -> tip of ocata branch build daily - the freshest
code that past gates
image-name:master -> tip of master branch

To achieve fully repeatable builds, we would need to have 4.0.0 type
tagging (based on pbr+git tag), version manifesto generation (as
discussed on PTG) and mechanism to consume existing manifestos and
rebuild images with these exact versions (baring issues with repo
removing previous version...). That is quite a project in it's own...

Thoughts?

Cheers,
Michal

On 17 April 2017 at 19:43, Jeffrey Zhang zhang.lei.fly@gmail.com wrote:
I think we have two topics and improvements here

  1. images in https://hub.docker.com/r/kolla/
  2. tag in end-user env.

images in hub.docker.com

we are building kolla tag image and push them into hub.docker.com. After
this,
we do nothing for these images.

The issue is

  1. any security update is not included in these images.
    solution: I do not think use 4.0.0-1 4.0.0-2 in hub.docker.com is a good
    idea.
    if so, we need mark what 4.0.0-1 container and what's the difference with
    4.0.0-2.
    This will make another chaos.
    And any prod env shouldn't depend on hub.docker.com's images, which is
    vulnerable
    to attack and is mutable.

  2. branch images are not pushed.
    solution: we can add a job to push branch images into hub.docker.com like
    inc0
    said. For example:
    centos-source-nova-api:4.0.0
    centos-source-nova-api:ocata
    centos-source-nova-api:pike
    centos-source-nova-api:master
    But branch tag images is not stable ( even its name is stable/ocata ),
    users are
    not recommended to use these images

images in end-user env

I recommended end user should build its own image rather then use
hub.docker.com directly.
in my env, I build images with following tag rule.

when using 4.0.0 to build multi time, i use different tag name. For example
1st: 4.0.0.1
2nd: 4.0.0.2
3rd: 4.0.0.3
...

The advantage in this way is: keep each tag as immutable ( never override )

On Tue, Apr 18, 2017 at 6:46 AM, Steve Baker sbaker@redhat.com wrote:

On Tue, Apr 18, 2017 at 9:57 AM, Doug Hellmann doug@doughellmann.com
wrote:

Excerpts from Michał Jastrzębski's message of 2017-04-12 15:59:34 -0700:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

Cheers,
Michal

What's in the images, kolla? Other OpenStack components?

Yes, each image will typically contain all software required for one
OpenStack service, including dependencies from OpenStack projects or the
base OS. Installed via some combination of git, pip, rpm, deb.

Where does the
4.0.0 come from?

Its the python version string from the kolla project itself, so ultimately
I think pbr. I'm suggesting that we switch to using the
version.release_string[1] which will tag with the longer version we use for
other dev packages.

[1]https://review.openstack.org/#/c/448380/1/kolla/common/config.py


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 18, 2017 by Michał_Jastrzębski (9,220 points)   1 5 5
0 votes

On 13/04/17 13:48 +1200, Steve Baker wrote:
On Thu, Apr 13, 2017 at 10:59 AM, Michał Jastrzębski inc007@gmail.com
wrote:

My dear Kollegues,

Today we had discussion about how to properly name/tag images being
pushed to dockerhub. That moved towards general discussion on revision
mgmt.

Problem we're trying to solve is this:
If you build/push images today, your tag is 4.0
if you do it tomorrow, it's still 4.0, and will keep being 4.0 until
we tag new release.

But image built today is not equal to image built tomorrow, so we
would like something like 4.0.0-1, 4.0.0-2.
While we can reasonably detect history of revisions in dockerhub,
local env will be extremely hard to do.

I'd like to ask you for opinions on desired behavior and how we want
to deal with revision management in general.

I already have a change which proposes tagging images with a pbr built
version [1]. I think if users want tags which are stable for the duration
of a major release they should switch to using the tag specified by
kolla-build.conf base_tag, which can be set to latest, ocata, pike, etc.
This would leave the version tag to at least track changes to the kolla
repo itself. Since the contents of upstream kolla images come from such
diverse sources, all I could suggest to ensure unique tags are created for
unique images is to append a datestamp to [1] (or have an extra datestamp
based tag). Bonus points for only publishing a new datestamp tag if the
contents of the image really changes.

In the RDO openstack-kolla package we now tag images with the
{Version}-{Release} of the openstack-kolla package which built it[2]. I
realise this doesn't solve the problem of the tag needing to change when
other image contents need to be updated, but I believe this can be solved
within the RDO image build pipeline by incrementing the {Release} whenever
a new image needs to be published.

[1] https://review.openstack.org/#/c/448380/
[2] https://review.rdoproject.org/r/#/c/5923/1/openstack-kolla.spec

I like this option better because it's more consistent with how things are done
elsewhere in OpenStack.

Flavio

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Apr 18, 2017 by Flavio_Percoco (36,960 points)   3 7 11
...