settingsLogin | Registersettings

[openstack-dev] [TripleO][Kolla] Concerns about containers images in DockerHub

0 votes

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image

docker_image module structure doesn't seem to encourage the addition of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 21, 2017 in openstack-dev by Gabriele_Cerami (760 points)   2

12 Responses

0 votes

For kolla, we were thinking about a couple of optimization that should greatly reduce the space.

  1. only upload to the hub based on stable versions. The updates are much less frequent.
  2. fingerprint the containers. base it on rpm/deb list, pip list, git checksums. If the fingerprint is the same, don't reupload a container. Nothing really changed but some trivial files or timestamps on files.

Also, remember the apparent size of a container is not the same as the actual size. Due to layering, the actual size is often significantly smaller then what shows up in 'docker images'. For example, this http://tarballs.openstack.org/kolla-kubernetes/gate/containers/centos-binary-ceph.tar.bz2 is only 1.2G and contains all the containers needed for a compute kit deployment.

For trunk based builds, it may still be a good idea to only mirror those to tarballs.o.o or a openstack provided docker repo that infra has been discussing?

Thanks,
Kevin


From: Gabriele Cerami [gcerami@redhat.com]
Sent: Thursday, October 19, 2017 8:03 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [TripleO][Kolla] Concerns about containers images in DockerHub

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image

docker_image module structure doesn't seem to encourage the addition of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be license
compatible for openstack. its structure should make it easily adapted to
delete an image. And you can copy it and cut it up thanks to the license.

Are you pushing images with no shared base layers at all (300MB compressed
image is no shared base layers)? With shared base layers a full image set
of Kolla images should be much smaller than the numbers you posted.

Thanks,
SamYaple

On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami gcerami@redhat.com
wrote:

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image

docker_image module structure doesn't seem to encourage the addition of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by Sam_Yaple (3,560 points)   3 3
0 votes

This reminded me of something I wanted to ask.

Is it true to state that only way to get 'fully' shared-base layers is
to have kolla-build build all the projects (that a
person/company/other may use) in one invocation? (in part because of the
jinja2 template generation which would cause differences in dockerfiles?...)

I was pretty sure this was the case (unless things have changed), but
just wanting to check since that question seems (somewhat) on-topic...

At godaddy we build individual projects using kolla-build (in part
because it makes it easy to rebuild + test + deply a single project with
either an update or a patch or ...) and I suspect others are doing this
also (after all the kolla-build command does take a regex of projects to
build) - though doing it in this way does seem like it would not reuse
(all the layers outside of the base operating system) layers 'optimally'?

Thoughts?

-Josh

Sam Yaple wrote:
dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be
license compatible for openstack. its structure should make it easily
adapted to delete an image. And you can copy it and cut it up thanks to
the license.

Are you pushing images with no shared base layers at all (300MB
compressed image is no shared base layers)? With shared base layers a
full image set of Kolla images should be much smaller than the numbers
you posted.

Thanks,
SamYaple

On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami <gcerami@redhat.com
gcerami@redhat.com> wrote:

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting a
   EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
   images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image
<https://docs.docker.com/registry/spec/api/#deleting-an-image>

docker_image module structure doesn't seem to encourage the addition of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by harlowja_at_fastmail (16,200 points)   2 5 8
0 votes

On 19 October 2017 at 13:32, Joshua Harlow harlowja@fastmail.com wrote:
This reminded me of something I wanted to ask.

Is it true to state that only way to get 'fully' shared-base layers is to
have kolla-build build all the projects (that a person/company/other may
use) in one invocation? (in part because of the jinja2 template generation
which would cause differences in dockerfiles?...)

Well jinja2 should render same dockerfile no matter when you call it,
so it should be fine. Alternatively you can run something like
kolla-build nova --skip-parents - this call will try to build all
images with "nova" in them while not rebuilding openstack-base and
base image.

I was pretty sure this was the case (unless things have changed), but just
wanting to check since that question seems (somewhat) on-topic...

At godaddy we build individual projects using kolla-build (in part because
it makes it easy to rebuild + test + deply a single project with either an
update or a patch or ...) and I suspect others are doing this also (after
all the kolla-build command does take a regex of projects to build) - though
doing it in this way does seem like it would not reuse (all the layers
outside of the base operating system) layers 'optimally'?

Thoughts?

-Josh

Sam Yaple wrote:

dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be
license compatible for openstack. its structure should make it easily
adapted to delete an image. And you can copy it and cut it up thanks to
the license.

Are you pushing images with no shared base layers at all (300MB
compressed image is no shared base layers)? With shared base layers a
full image set of Kolla images should be much smaller than the numbers
you posted.

Thanks,
SamYaple

On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami <gcerami@redhat.com
gcerami@redhat.com> wrote:

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly

upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting

a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and

to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image
<https://docs.docker.com/registry/spec/api/#deleting-an-image>

docker_image module structure doesn't seem to encourage the addition

of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by Michał_Jastrzębski (9,220 points)   1 4 5
0 votes

On 19 October 2017 at 13:37, Michał Jastrzębski inc007@gmail.com wrote:
On 19 October 2017 at 13:32, Joshua Harlow harlowja@fastmail.com wrote:

This reminded me of something I wanted to ask.

Is it true to state that only way to get 'fully' shared-base layers is to
have kolla-build build all the projects (that a person/company/other may
use) in one invocation? (in part because of the jinja2 template generation
which would cause differences in dockerfiles?...)

Well jinja2 should render same dockerfile no matter when you call it,
so it should be fine. Alternatively you can run something like
kolla-build nova --skip-parents - this call will try to build all
images with "nova" in them while not rebuilding openstack-base and
base image.

I was pretty sure this was the case (unless things have changed), but just
wanting to check since that question seems (somewhat) on-topic...

At godaddy we build individual projects using kolla-build (in part because
it makes it easy to rebuild + test + deply a single project with either an
update or a patch or ...) and I suspect others are doing this also (after
all the kolla-build command does take a regex of projects to build) - though
doing it in this way does seem like it would not reuse (all the layers
outside of the base operating system) layers 'optimally'?

Thoughts?

-Josh

Sam Yaple wrote:

dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be
license compatible for openstack. its structure should make it easily
adapted to delete an image. And you can copy it and cut it up thanks to
the license.

Are you pushing images with no shared base layers at all (300MB
compressed image is no shared base layers)? With shared base layers a
full image set of Kolla images should be much smaller than the numbers
you posted.

Thanks,
SamYaple

On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami <gcerami@redhat.com
gcerami@redhat.com> wrote:

Hi,

our CI scripts are now automatically building, testing and pushing
approved openstack/RDO services images to public repositories in
dockerhub using ansible docker_image module.

Promotions have had some hiccups, but we're starting to regularly

upload
new images every 4 hours.

When we'll get at full speed, we'll potentially have
- 3-4 different sets of images, one per release of openstack (counting

a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

At around 300MB per image a possible grand total is around 650GB of
space used.

That doesn't sound correct as images share a lot - full registry of
single type/distro (like centos source) is ~10gig

We don't know if this is acceptable usage of dockerhub space and for
this we already sent a similar email the to docker support to ask
specifically if the user would get penalty in any way (e.g. enforcing
quotas, rete limiting, blocking). We're still waiting for a reply.

In any case it's critical to keep the usage around the estimate, and

to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

https://docs.docker.com/registry/spec/api/#deleting-an-image
<https://docs.docker.com/registry/spec/api/#deleting-an-image>

docker_image module structure doesn't seem to encourage the addition

of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

We'd appreciate any input for an existing, in progress and/or better
solution for bulk deletion, and issues that may arise with our space
usage in dockerhub

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by Michał_Jastrzębski (9,220 points)   1 4 5
0 votes

Cool thanks,

I'll have to watch for this and see how it goes.

I thought there was some specific things that changed every time a
dockerfile was rendered (which would make it different each run) but I
may have been seeing things (or it's been fixed).

Michał Jastrzębski wrote:
On 19 October 2017 at 13:32, Joshua Harlowharlowja@fastmail.com wrote:

This reminded me of something I wanted to ask.

Is it true to state that only way to get 'fully' shared-base layers is to
have kolla-build build all the projects (that a person/company/other may
use) in one invocation? (in part because of the jinja2 template generation
which would cause differences in dockerfiles?...)

Well jinja2 should render same dockerfile no matter when you call it,
so it should be fine. Alternatively you can run something like
kolla-build nova --skip-parents - this call will try to build all
images with "nova" in them while not rebuilding openstack-base and
base image.

I was pretty sure this was the case (unless things have changed), but just
wanting to check since that question seems (somewhat) on-topic...

At godaddy we build individual projects using kolla-build (in part because
it makes it easy to rebuild + test + deply a single project with either an
update or a patch or ...) and I suspect others are doing this also (after
all the kolla-build command does take a regex of projects to build) - though
doing it in this way does seem like it would not reuse (all the layers
outside of the base operating system) layers 'optimally'?

Thoughts?

-Josh

Sam Yaple wrote:

dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be
license compatible for openstack. its structure should make it easily
adapted to delete an image. And you can copy it and cut it up thanks to
the license.

Are you pushing images with no shared base layers at all (300MB
compressed image is no shared base layers)? With shared base layers a
full image set of Kolla images should be much smaller than the numbers
you posted.

Thanks,
SamYaple

On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami<gcerami@redhat.com
gcerami@redhat.com> wrote:

 Hi,

 our CI scripts are now automatically building, testing and pushing
 approved openstack/RDO services images to public repositories in
 dockerhub using ansible docker_image module.

 Promotions have had some hiccups, but we're starting to regularly

upload
new images every 4 hours.

 When we'll get at full speed, we'll potentially have
 - 3-4 different sets of images, one per release of openstack (counting

a
EOL release grace period)
- 90-100 different services images per release
- 4-6 different versions of the same image ( keeping older promoted
images for a while )

 At around 300MB per image a possible grand total is around 650GB of
 space used.

 We don't know if this is acceptable usage of dockerhub space and for
 this we already sent a similar email the to docker support to ask
 specifically if the user would get penalty in any way (e.g. enforcing
 quotas, rete limiting, blocking). We're still waiting for a reply.

 In any case it's critical to keep the usage around the estimate, and

to
achieve this we need a way to automatically delete the older images.
docker_image module does not provide this functionality, and we think
the only way is issuing direct calls to dockerhub API

 https://docs.docker.com/registry/spec/api/#deleting-an-image
 <https://docs.docker.com/registry/spec/api/#deleting-an-image>

 docker_image module structure doesn't seem to encourage the addition

of
such functionality directly in it, so we may be forced to use the uri
module.
With new images uploaded potentially every 4 hours, this will become a
problem to be solved within the next two weeks.

 We'd appreciate any input for an existing, in progress and/or better
 solution for bulk deletion, and issues that may arise with our space
 usage in dockerhub

 Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 19, 2017 by harlowja_at_fastmail (16,200 points)   2 5 8
0 votes

On 19 Oct, Sam Yaple wrote:
dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be license
compatible for openstack. its structure should make it easily adapted to
delete an image. And you can copy it and cut it up thanks to the license.

Thanks, I'll look into it.

Are you pushing images with no shared base layers at all (300MB compressed
image is no shared base layers)? With shared base layers a full image set
of Kolla images should be much smaller than the numbers you posted.

300MB is the rounded size of the size reported by the dockerhub UI
e.g https://hub.docker.com/r/tripleopike/centos-binary-heat-api/
shows 265MB for the newest tag. I'm not sure what size is dockerhub
reporting.

When pulling the image, docker downloads 30 layers. The final size
reported locally is 815MB.

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Gabriele_Cerami (760 points)   2
0 votes

On Thu, Oct 19, 2017 at 9:38 PM, Gabriele Cerami gcerami@redhat.com wrote:

On 19 Oct, Sam Yaple wrote:

dockerimage wouldn't be the best place for that. Buf if you are looking
for a quicker solution, kolla
docker was written specifically to be
license
compatible for openstack. its structure should make it easily adapted to
delete an image. And you can copy it and cut it up thanks to the license.

Thanks, I'll look into it.

Are you pushing images with no shared base layers at all (300MB
compressed
image is no shared base layers)? With shared base layers a full image set
of Kolla images should be much smaller than the numbers you posted.

300MB is the rounded size of the size reported by the dockerhub UI
e.g https://hub.docker.com/r/tripleopike/centos-binary-heat-api/
shows 265MB for the newest tag. I'm not sure what size is dockerhub
reporting.

This is misleading. For example, you will download 265MB if you download
only tripleopike/centos-binary-heat-api:current-tripleo . But if you
download both tripleopike/centos-binary-heat-api:current-tripleo and
tripleopike/centos-binary-heat-engine:current-tripleo you will have only
downloaded 266MB in total since the majority of those layers are shared.

So it seems tripleo is building all images and then pushing them.
Reworking your number leads me to believe you will be consuming 10-15GB in
total on Dockerhub. Kolla images are only the size that you posted when
built as seperate services. Just keep building all the images at the same
time and you wont get anywhere near the numbers you posted.

When pulling the image, docker downloads 30 layers. The final size
reported locally is 815MB.

This is the uncompressed size, but even here layers are shared.

Thanks


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Sam_Yaple (3,560 points)   3 3
0 votes

On 19 Oct, Sam Yaple wrote:
So it seems tripleo is building all images and then pushing them.
Reworking your number leads me to believe you will be consuming 10-15GB in
total on Dockerhub. Kolla images are only the size that you posted when
built as seperate services. Just keep building all the images at the same
time and you wont get anywhere near the numbers you posted.

Makes sense, so considering the shared layers
- a size of 10-15GB per build.
- 4-6 builds rotated per release
- 3-4 releases

total size will be approximately be 360GB in the worst case, and 120GB in
the best case, which seems a bit more reasonable.

Thanks for he clarifications


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Gabriele_Cerami (760 points)   2
0 votes

On Thu, Oct 19, 2017 at 11:23 PM, Gabriele Cerami gcerami@redhat.com
wrote:

On 19 Oct, Sam Yaple wrote:

So it seems tripleo is building all images and then pushing them.
Reworking your number leads me to believe you will be consuming 10-15GB
in
total on Dockerhub. Kolla images are only the size that you posted when
built as seperate services. Just keep building all the images at the same
time and you wont get anywhere near the numbers you posted.

Makes sense, so considering the shared layers
- a size of 10-15GB per build.
- 4-6 builds rotated per release
- 3-4 releases

  • a size of 1-2GB per build
  • 4-6 builds rotated per release
  • 3-4 releases

At worst you are looking at 48GB not 360GB. Dont worry so much there!

total size will be approximately be 360GB in the worst case, and 120GB in
the best case, which seems a bit more reasonable.

Thanks for he clarifications


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 20, 2017 by Sam_Yaple (3,560 points)   3 3
...