settingsLogin | Registersettings

[openstack-dev] Glance Store Future

0 votes

cc'ing ML since it's an important discussion, IMO...

On 07/31/2014 11:54 AM, Arnaud Legendre wrote:

Hi Jay,

I would be interested if you could share your point of view on this
item: we want to make the glance stores a standalone library
(glance.stores) which would be consumed directly by Nova and Cinder.

Yes, I have been enthusiastic about this effort for a long time now :)
In fact, I have been pushing a series of patches (most merged at this
point) in Nova to clean up the (very) messy nova.image.glance module and
standardize the image API in Nova.

The messiest part of the current image API in Nova, by far, is the
nova.image.glance.GlanceImageService.download() method, which you
highlight below. The reason it is so messy is that the method does
different things (and returns different things!) depending on how you
call it and what arguments you provide. :(

I think it would be nice to get your pov since you worked a lot on
the Nova image interface recently. To give you an example:

Here
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L333,
we would do:

  1. location = getimagelocation(image_id),
  2. get(location) from the
    glance.stores library like for example rbd
    (https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L206)

Yup. Though I'd love for this code to live in olso, not glance...

Plus, I'd almost prefer to see an interface that hides the location URIs
entirely and makes the discovery of those location URIs entirely
encapsulated within glance.store. So, for instance, instead of getting
the image location using a call to glanceclient.show(), parsing the
locations collection from the v2 API response, and passing that URI to
the glance.store.get() function, I'd prefer to see an interface more
like this:

```python

This code would go in a new nova.image.API.copy() method:

import io

from oslo.image import move
from oslo.image.move import exception as mexc

from nova import exception as exc

...
def copy(imageidoruri, streamwriter):
try:
config = {
# Some Nova CONF options...
}
mover = move.Mover(imageidoruri, config)
success, bytes
written = mover.copy(streamwriter)
if success:
if bytes
written == 0:
LOG.info("Copied image %s using zero-copy "
"transfer.", imageidoruri)
else:
LOG.info("Copied image %s using standard "
"filesystem copy. Copied %d bytes.",
image
idoruri, byteswritten)
return success
except mexc.ImageNotFound:
raise exc.NotFound(...)
except mexc.ImageInvalidApi:
# Fall back to pull image from Glance
# API server via HTTP and write to disk
# via the stream
writer argument's write()
# interface... and return True or False
# depending on whether write()s succeeded
```

And then, the caller of such an nova.image.API.copy() function would be
in the existing various virt utils and imagebackends, and would call the
API function like so:

```python

This code would go in something like nova.virt.libvirt.utils:

from nova import image

IMAGE_API = image.API()

writefile = io.FileIO(dstpath, mode='wb')
writer = io.BufferedWriter(write_file)

imageidor_uri = "https://images.example.com/images/123"

result = IMAGEAPI.copy(imageidoruri, writer)

Test result if needed...

```

Notice that the caller never needs to know about the locations
collection of the image -- and thus we correct the leaked implementation
details that currently ooze out of the download() method in
nova.image.glance.GlanceImageService.download.

Also note that we no longer pass a variety of file descriptors, file
writers, file destination paths to the download method. Instead, we
always just pass the image ID or URI and a writeable bytestream
iterator. And we always return either True or False, instead of None or
a file iterator depending on the supplied arguments to download().

The same kind of logic could be added in Cinder.

Sure.

We see that as a benefit for Nova, which would be able to directly
consume the stores instead of going through the glance api.

Exactly.

We had a vote today to figure out if we continue the effort on the
glance.stores library. We had a majority of +1 but there was a couple
of -1 due to the fact that we don?t have enough concrete examples of
this will be useful or not.

It will definitely be useful in the following:

1) Making the copy/zero-copy/transfer/download methods consistent
between all the various places in the Nova virt drivers that do similar
things.

2) Allowing a single place to innovate for the transfer of image bits
between sources and destinations

Hopefully, the above sample code and interfaces will spark some renewed
interest in this. I'd love to work on it, and see it pushed forward.

Best,
-jay

Please also see this etherpad containing a list of
advantages/drawbacks: https://etherpad.openstack.org/p/glance-store

Looking forward to get your pov.

Thanks, Arnaud

On Jul 28, 2014, at 1:23 PM, Flavio Percoco > wrote:

On 07/28/2014 09:12 PM, Arnaud Legendre wrote:

Flavio ? yeah, glance.store lost momentum at the meetup? I think
it would be nice if you could recap the advantages of doing
that. At the meetup, we quickly did an "advantages vs drawbacks?
poll and I think we might have missed advantages (since you were
not here :)) so there was more drawbacks.

Was this published somewhere?

Also, the gap between glance.store and glance will continuously
increase if we do not define some rules to keep both in sync
(while we are making glance.store ready). If we choose to use
glance.store, I think we have to switch glance to use it asap.

It's not difficult to keep the gap close if we keep enough eyes on
reviews. We've been doing this for oslo.messaging and the old
oslo-rpc code.

This has been added to the agenda for the irc meeting next
Thursday, that would be awesome if you could attend...

I'll try to make it there but if I can't, I'd really appreciate
this discussion to be brought to the mailing list. I'd really like
to understand why the feeling changed and what are the bases for
this new feeling about the library.

Pls, don't get me wrong. I'm just really surprised and a bit
frustrated but I do want to understand what the current feeling
about it is and hopefully convince the team otherwise.

Cheers, Flavio

-Arnaud

On Jul 28, 2014, at 9:44 AM, Flavio Percoco > wrote:

Hey Guys,

I was talking to Nikhil earlier today and he mentioned there
was a mini discussion about glance.store in the mini-summit.
One thing he mentioned is that some folks don't want it to be
merged/exists.

I'm writing to get your feeling about this and understand what
the concerns are. After 1 1/2 development cycle working on the
library, based on our previous agreement, I'm sure you guys
understand my frustration after hearing this.

Mark, I understand you are going on vacation in about 2 weeks,
I'd really like to clear this before you leave so I can move
this work forward and don't miss the feature freeze.

Thanks in advance for any feedback, Flavio

P.S: If you guys prefer, we can have a meeting about me. FWIW,
I've been trying to contact most of you on IRC but it's been
really hard because of the timezones.

-- @flaper87 Flavio Percoco

-- @flaper87 Flavio Percoco

asked Aug 1, 2014 in openstack-dev by Jay_Pipes (59,760 points)   3 11 15
retagged Feb 25, 2015 by admin

8 Responses

0 votes

Two inline replies, throw out my quick thougths.

On Sat, Aug 2, 2014 at 4:41 AM, Jay Pipes wrote:
cc'ing ML since it's an important discussion, IMO...

On 07/31/2014 11:54 AM, Arnaud Legendre wrote:

Hi Jay,

I would be interested if you could share your point of view on this
item: we want to make the glance stores a standalone library
(glance.stores) which would be consumed directly by Nova and Cinder.

Yes, I have been enthusiastic about this effort for a long time now :) In
fact, I have been pushing a series of patches (most merged at this point) in
Nova to clean up the (very) messy nova.image.glance module and standardize
the image API in Nova.

The messiest part of the current image API in Nova, by far, is the
nova.image.glance.GlanceImageService.download() method, which you highlight
below. The reason it is so messy is that the method does different things
(and returns different things!) depending on how you call it and what
arguments you provide. :(

I think it would be nice to get your pov since you worked a lot on
the Nova image interface recently. To give you an example:

Here
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L333,
we would do:

  1. location = getimagelocation(image_id),
  2. get(location) from the
    glance.stores library like for example rbd
    (https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L206)

Yup. Though I'd love for this code to live in olso, not glance...

Plus, I'd almost prefer to see an interface that hides the location URIs
entirely and makes the discovery of those location URIs entirely
encapsulated within glance.store. So, for instance, instead of getting the
image location using a call to glanceclient.show(), parsing the locations
collection from the v2 API response, and passing that URI to the
glance.store.get() function, I'd prefer to see an interface more like this:

```python

This code would go in a new nova.image.API.copy() method:

import io

from oslo.image import move
from oslo.image.move import exception as mexc

from nova import exception as exc

...
def copy(imageidoruri, streamwriter):
try:
config = {
# Some Nova CONF options...
}
mover = move.Mover(imageidoruri, config)
success, bytes
written = mover.copy(streamwriter)
if success:
if bytes
written == 0:
LOG.info("Copied image %s using zero-copy "
"transfer.", imageidoruri)
else:
LOG.info("Copied image %s using standard "
"filesystem copy. Copied %d bytes.",
image
idoruri, byteswritten)
return success
except mexc.ImageNotFound:
raise exc.NotFound(...)
except mexc.ImageInvalidApi:
# Fall back to pull image from Glance
# API server via HTTP and write to disk
# via the stream
writer argument's write()
# interface... and return True or False
# depending on whether write()s succeeded
```

This idea looks more neat, but I'm a little worries on implementation
since most CoW based zero-copy and smart full-copy approaches need
leverage the capability from particular storage (e.g. ceph) and/or
hypervisor (e.g. vsphere), so IMHO we almost couldn't to make a
consistent logic of zero-copy or smart full-copy/transferring and
encapsulate them into glance.store (or oslo.image) that separated from
special hypervisor and storage context, only if we do necessary
function that hypervisor and storage needed in the lib internal.

And then, the caller of such an nova.image.API.copy() function would be in
the existing various virt utils and imagebackends, and would call the API
function like so:

```python

This code would go in something like nova.virt.libvirt.utils:

from nova import image

IMAGE_API = image.API()

writefile = io.FileIO(dstpath, mode='wb')
writer = io.BufferedWriter(write_file)

imageidor_uri = "https://images.example.com/images/123"

result = IMAGEAPI.copy(imageidoruri, writer)

Test result if needed...

```

Notice that the caller never needs to know about the locations collection of
the image -- and thus we correct the leaked implementation details that
currently ooze out of the download() method in
nova.image.glance.GlanceImageService.download.

Also note that we no longer pass a variety of file descriptors, file
writers, file destination paths to the download method. Instead, we always
just pass the image ID or URI and a writeable bytestream iterator. And we
always return either True or False, instead of None or a file iterator
depending on the supplied arguments to download().

The same kind of logic could be added in Cinder.

Sure.

We see that as a benefit for Nova, which would be able to directly
consume the stores instead of going through the glance api.

Exactly.

We had a vote today to figure out if we continue the effort on the
glance.stores library. We had a majority of +1 but there was a couple
of -1 due to the fact that we don?t have enough concrete examples of
this will be useful or not.

It will definitely be useful in the following:

1) Making the copy/zero-copy/transfer/download methods consistent between
all the various places in the Nova virt drivers that do similar things.

2) Allowing a single place to innovate for the transfer of image bits
between sources and destinations

I agreed zero-copy and smart full-copy/transferring functions are
worth cases for us, but IMHO, at least from current basic
understanding, they seems don't like a responsibility of glance.store
lib, currently the lib just provides the basic IO function set for
those necessary operations backend storage direct related on image
bits (including add, delete, getsize, g/setacls), so even I'm sure
we can add more interfaces to glance.store to support more innovatory
functions in future, I'm afraid if it's a good place to place those
high level functions instead of a separated (new or existing)
service/lib.

Thanks,
zhiyan

Hopefully, the above sample code and interfaces will spark some renewed
interest in this. I'd love to work on it, and see it pushed forward.

Best,
-jay

Please also see this etherpad containing a list of
advantages/drawbacks: https://etherpad.openstack.org/p/glance-store

Looking forward to get your pov.

Thanks, Arnaud

On Jul 28, 2014, at 1:23 PM, Flavio Percoco > wrote:

On 07/28/2014 09:12 PM, Arnaud Legendre wrote:

Flavio ? yeah, glance.store lost momentum at the meetup? I think
it would be nice if you could recap the advantages of doing
that. At the meetup, we quickly did an "advantages vs drawbacks?
poll and I think we might have missed advantages (since you were
not here :)) so there was more drawbacks.

Was this published somewhere?

Also, the gap between glance.store and glance will continuously
increase if we do not define some rules to keep both in sync
(while we are making glance.store ready). If we choose to use
glance.store, I think we have to switch glance to use it asap.

It's not difficult to keep the gap close if we keep enough eyes on
reviews. We've been doing this for oslo.messaging and the old
oslo-rpc code.

This has been added to the agenda for the irc meeting next
Thursday, that would be awesome if you could attend...

I'll try to make it there but if I can't, I'd really appreciate
this discussion to be brought to the mailing list. I'd really like
to understand why the feeling changed and what are the bases for
this new feeling about the library.

Pls, don't get me wrong. I'm just really surprised and a bit
frustrated but I do want to understand what the current feeling
about it is and hopefully convince the team otherwise.

Cheers, Flavio

-Arnaud

On Jul 28, 2014, at 9:44 AM, Flavio Percoco > wrote:

Hey Guys,

I was talking to Nikhil earlier today and he mentioned there
was a mini discussion about glance.store in the mini-summit.
One thing he mentioned is that some folks don't want it to be
merged/exists.

I'm writing to get your feeling about this and understand what
the concerns are. After 1 1/2 development cycle working on the
library, based on our previous agreement, I'm sure you guys
understand my frustration after hearing this.

Mark, I understand you are going on vacation in about 2 weeks,
I'd really like to clear this before you leave so I can move
this work forward and don't miss the feature freeze.

Thanks in advance for any feedback, Flavio

P.S: If you guys prefer, we can have a meeting about me. FWIW,
I've been trying to contact most of you on IRC but it's been
really hard because of the timezones.

-- @flaper87 Flavio Percoco

-- @flaper87 Flavio Percoco

responded Aug 4, 2014 by Zhi_Yan_Liu (1,300 points)   1 2
0 votes

On 08/01/2014 10:41 PM, Jay Pipes wrote:
cc'ing ML since it's an important discussion, IMO...

On 07/31/2014 11:54 AM, Arnaud Legendre wrote:

Hi Jay,

I would be interested if you could share your point of view on this
item: we want to make the glance stores a standalone library
(glance.stores) which would be consumed directly by Nova and Cinder.

Yes, I have been enthusiastic about this effort for a long time now :)
In fact, I have been pushing a series of patches (most merged at this
point) in Nova to clean up the (very) messy nova.image.glance module and
standardize the image API in Nova.

The messiest part of the current image API in Nova, by far, is the
nova.image.glance.GlanceImageService.download() method, which you
highlight below. The reason it is so messy is that the method does
different things (and returns different things!) depending on how you
call it and what arguments you provide. :(

+1

I think it would be nice to get your pov since you worked a lot on
the Nova image interface recently. To give you an example:

Here
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L333,
we would do:

  1. location = getimagelocation(image_id),
  2. get(location) from the
    glance.stores library like for example rbd
    (https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L206)

Yup. Though I'd love for this code to live in olso, not glance...

I think both places make sense. The reason why we decided to keep it
under glance is because it's still an important piece of the project and
the team is willing to maintain it, plus the team is already familiar
with the code.

Not sure how strong those points are but AFAIR, those two are the reason
why the lib lives under glance. Here's the link to the email thread were
this was discussed:

http://lists.openstack.org/pipermail/openstack-dev/2013-December/022907.html

Plus, I'd almost prefer to see an interface that hides the location URIs
entirely and makes the discovery of those location URIs entirely
encapsulated within glance.store. So, for instance, instead of getting
the image location using a call to glanceclient.show(), parsing the
locations collection from the v2 API response, and passing that URI to
the glance.store.get() function, I'd prefer to see an interface more
like this:

FWIW, The API is not finished (probably not even started :D). The first
step we wanted to pursue was pulling the code out of Glance and just
then work on an improved, more secure and more consistent API. Your
proposal looks neat, I'll propose a design session for the glance.store
API. :D

Thanks for your feedback,
Flavio

--
@flaper87
Flavio Percoco

responded Aug 4, 2014 by Flavio_Percoco (36,960 points)   3 8 14
0 votes

Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things into
Oslo, but our (cinder team) general experiences with the end result have
been highly variable, to the point where we've discussed just saying no to
Oslo code since the pain is more than the gain. In this case, the glance
team are the subject matter experts, the glance interfaces and internals
are their core competency, I honestly can't see any value in putting the
project anywhere other than glance
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Aug 4, 2014 by Duncan_Thomas (16,160 points)   1 3 6
0 votes

On 08/04/2014 09:09 AM, Duncan Thomas wrote:
Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" > wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things into
Oslo, but our (cinder team) general experiences with the end result have
been highly variable, to the point where we've discussed just saying no
to Oslo code since the pain is more than the gain. In this case, the
glance team are the subject matter experts, the glance interfaces and
internals are their core competency, I honestly can't see any value in
putting the project anywhere other than glance

2 reasons.

1) This is code that will be utilized by >1 project, and is a library,
not a service endpoint. That seems to be right up the Oslo alley.

2) The mission of the Glance program has changed to being an application
catalog service, not an image streaming service.

Best,
-jay

responded Aug 4, 2014 by Jay_Pipes (59,760 points)   3 11 15
0 votes

On Aug 4, 2014, at 9:27 AM, Jay Pipes wrote:

On 08/04/2014 09:09 AM, Duncan Thomas wrote:

Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" > wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things into
Oslo, but our (cinder team) general experiences with the end result have
been highly variable, to the point where we've discussed just saying no
to Oslo code since the pain is more than the gain. In this case, the
glance team are the subject matter experts, the glance interfaces and
internals are their core competency, I honestly can't see any value in
putting the project anywhere other than glance

2 reasons.

1) This is code that will be utilized by >1 project, and is a library, not a service endpoint. That seems to be right up the Oslo alley.

2) The mission of the Glance program has changed to being an application catalog service, not an image streaming service.

Best,
-jay

Oslo isn?t the only program that can produce reusable libraries, though. If the Glance team is going to manage this code anyway, it makes sense to leave it in the Glance program.

Doug

responded Aug 4, 2014 by Doug_Hellmann (87,520 points)   4 5 13
0 votes

On 08/04/2014 09:46 AM, Doug Hellmann wrote:

On Aug 4, 2014, at 9:27 AM, Jay Pipes wrote:

On 08/04/2014 09:09 AM, Duncan Thomas wrote:

Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" > wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things into
Oslo, but our (cinder team) general experiences with the end result have
been highly variable, to the point where we've discussed just saying no
to Oslo code since the pain is more than the gain. In this case, the
glance team are the subject matter experts, the glance interfaces and
internals are their core competency, I honestly can't see any value in
putting the project anywhere other than glance

2 reasons.

1) This is code that will be utilized by >1 project, and is a library, not a service endpoint. That seems to be right up the Oslo alley.

2) The mission of the Glance program has changed to being an application catalog service, not an image streaming service.

Best,
-jay

Oslo isn?t the only program that can produce reusable libraries, though. If the Glance team is going to manage this code anyway, it makes sense to leave it in the Glance program.

Agreed. Honestly it's better to keep the review teams close to the
expertise for the function at hand.

It needs to be ok that teams besides oslo create reusable components for
other parts of OpenStack. Oslo should be used for things where there
isn't a strong incumbent owner. I think we have a strong incumbent owner
here so living in Artifacts program makes sense to me.

-Sean

--
Sean Dague
http://dague.net

responded Aug 4, 2014 by Sean_Dague (66,200 points)   4 11 21
0 votes

On 08/04/2014 10:29 AM, Sean Dague wrote:
On 08/04/2014 09:46 AM, Doug Hellmann wrote:

On Aug 4, 2014, at 9:27 AM, Jay Pipes wrote:

On 08/04/2014 09:09 AM, Duncan Thomas wrote:

Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" > wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things into
Oslo, but our (cinder team) general experiences with the end result have
been highly variable, to the point where we've discussed just saying no
to Oslo code since the pain is more than the gain. In this case, the
glance team are the subject matter experts, the glance interfaces and
internals are their core competency, I honestly can't see any value in
putting the project anywhere other than glance

2 reasons.

1) This is code that will be utilized by >1 project, and is a library, not a service endpoint. That seems to be right up the Oslo alley.

2) The mission of the Glance program has changed to being an application catalog service, not an image streaming service.

Best,
-jay

Oslo isn?t the only program that can produce reusable libraries, though. If the Glance team is going to manage this code anyway, it makes sense to leave it in the Glance program.

Agreed. Honestly it's better to keep the review teams close to the
expertise for the function at hand.

It needs to be ok that teams besides oslo create reusable components for
other parts of OpenStack. Oslo should be used for things where there
isn't a strong incumbent owner. I think we have a strong incumbent owner
here so living in Artifacts program makes sense to me.

Sure, fair points from all. If it can be imported/packaged without
including all the legacy Glance code, then I'd be more behind keeping it
in Glance...

-jay

responded Aug 4, 2014 by Jay_Pipes (59,760 points)   3 11 15
0 votes

On 08/04/2014 05:56 PM, Jay Pipes wrote:
On 08/04/2014 10:29 AM, Sean Dague wrote:

On 08/04/2014 09:46 AM, Doug Hellmann wrote:

On Aug 4, 2014, at 9:27 AM, Jay Pipes wrote:

On 08/04/2014 09:09 AM, Duncan Thomas wrote:

Duncan Thomas
On Aug 1, 2014 9:44 PM, "Jay Pipes" > wrote:

Yup. Though I'd love for this code to live in olso, not glance...

Why Oslo? There seems to be a general obsession with getting things
into
Oslo, but our (cinder team) general experiences with the end result
have
been highly variable, to the point where we've discussed just
saying no
to Oslo code since the pain is more than the gain. In this case, the
glance team are the subject matter experts, the glance interfaces and
internals are their core competency, I honestly can't see any value in
putting the project anywhere other than glance

2 reasons.

1) This is code that will be utilized by >1 project, and is a
library, not a service endpoint. That seems to be right up the Oslo
alley.

2) The mission of the Glance program has changed to being an
application catalog service, not an image streaming service.

Best,
-jay

Oslo isn?t the only program that can produce reusable libraries,
though. If the Glance team is going to manage this code anyway, it
makes sense to leave it in the Glance program.

Agreed. Honestly it's better to keep the review teams close to the
expertise for the function at hand.

It needs to be ok that teams besides oslo create reusable components for
other parts of OpenStack. Oslo should be used for things where there
isn't a strong incumbent owner. I think we have a strong incumbent owner
here so living in Artifacts program makes sense to me.

Sure, fair points from all. If it can be imported/packaged without
including all the legacy Glance code, then I'd be more behind keeping it
in Glance...

FWIW, it's already like that. I'm working on the Glance port now[0],
which passes locally but not in the gate due to glance.store not being
released yet.

[0] https://review.openstack.org/#/c/100636/

Cheers,
Flavio

--
@flaper87
Flavio Percoco

responded Aug 4, 2014 by Flavio_Percoco (36,960 points)   3 8 14
...