settingsLogin | Registersettings

[Openstack-operators] what is the different in use Qcow2 or Raw in Ceph

0 votes

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder, nova
and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked May 28, 2015 in openstack-operators by Shake_Chen (1,460 points)   1 2 4

16 Responses

0 votes

The primary difference is the ability for CEPH to make zero byte copies.
When you use qcow2, ceph must actually create a complete copy instead of a
zero byte copy as it cannot do its own copy-on-write tricks with a qcow2
image.

So, yes, it will work fine with qcow2 images but it won't be as performant
as it is with RAW. Also, it will actually use more of the native underlying
storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder, nova
and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by David_Medberry (8,000 points)   1 4 5
0 votes

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry openstack@medberry.net
wrote:

The primary difference is the ability for CEPH to make zero byte copies.
When you use qcow2, ceph must actually create a complete copy instead of a
zero byte copy as it cannot do its own copy-on-write tricks with a qcow2
image.

So, yes, it will work fine with qcow2 images but it won't be as performant
as it is with RAW. Also, it will actually use more of the native underlying
storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder, nova
and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by David_Medberry (8,000 points)   1 4 5
0 votes

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can Ceph
be useful if snapshotting isn't an option with raw images and qcow2 isn't
recommended? Are there other factors to take into consideration that I'm
missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry openstack@medberry.net
wrote:

The primary difference is the ability for CEPH to make zero byte copies.
When you use qcow2, ceph must actually create a complete copy instead of a
zero byte copy as it cannot do its own copy-on-write tricks with a qcow2
image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by Forrest_Flagg (320 points)   1 3
0 votes

This isn't remotely related to btrfs. It works fine with XFS. Not sure how
that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.flagg@gmail.com
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can Ceph
be useful if snapshotting isn't an option with raw images and qcow2 isn't
recommended? Are there other factors to take into consideration that I'm
missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry openstack@medberry.net
wrote:

The primary difference is the ability for CEPH to make zero byte copies.
When you use qcow2, ceph must actually create a complete copy instead of a
zero byte copy as it cannot do its own copy-on-write tricks with a qcow2
image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by David_Medberry (8,000 points)   1 4 5
0 votes

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry openstack@medberry.net
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure how
that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.flagg@gmail.com
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and qcow2
isn't recommended? Are there other factors to take into consideration that
I'm missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry openstack@medberry.net
wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete copy
instead of a zero byte copy as it cannot do its own copy-on-write tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by Stephen_Cousins (380 points)   1
0 votes

yep. It's at the CEPH level (not the XFS level.)

On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins steve.cousins@maine.edu
wrote:

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry openstack@medberry.net
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure
how that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.flagg@gmail.com
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and qcow2
isn't recommended? Are there other factors to take into consideration that
I'm missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry <openstack@medberry.net

wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete copy
instead of a zero byte copy as it cannot do its own copy-on-write tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by David_Medberry (8,000 points)   1 4 5
0 votes

David is right, Ceph implements volume snapshotting at the RBD level,
not even RADOS level: whole 2 levels of abstraction above file system.
It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
VFAT): Ceph RBD takes care of it before individual chunks of an RBD
volume are passed to RADOS as objects and get written into the file
system as files by an OSD process.

The reason Fuel documentation recommends to disable QCOW2 format for
images is that RBD does not support QCOW2 disks directly, so Nova and
Cinder have to convert a QCOW2 image into RAW format before passing
it to QEMU's RBD driver. This means that you end up downloading the
QCOW2 image from Ceph to a nova-compute node (first full copy),
converting it (second full copy), and uploading the resultant RAW
image back to Ceph (third full copy) just to launch a VM or create a
volume from an image.

On Thu, May 28, 2015 at 8:33 AM, David Medberry openstack@medberry.net wrote:
yep. It's at the CEPH level (not the XFS level.)

On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins steve.cousins@maine.edu
wrote:

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry openstack@medberry.net
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure
how that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.flagg@gmail.com
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and qcow2
isn't recommended? Are there other factors to take into consideration that
I'm missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry
openstack@medberry.net wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete copy
instead of a zero byte copy as it cannot do its own copy-on-write tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Dmitry Borodaenko


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by Dmitry_Borodaenko (7,240 points)   1 3 5
0 votes

I've experienced the opposite problem though. Downloading raw images and uploading them to the cloud is very slow. Doing it through qcow2 allows them to be compressed over the slow links. Ideally, the Ceph driver would take a qcow2 and convert it to raw on glance ingest rather then at boot.

Thanks,
Kevin


From: Dmitry Borodaenko [dborodaenko@mirantis.com]
Sent: Thursday, May 28, 2015 12:10 PM
To: David Medberry
Cc: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] what is the different in use Qcow2 or Raw in Ceph

David is right, Ceph implements volume snapshotting at the RBD level,
not even RADOS level: whole 2 levels of abstraction above file system.
It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
VFAT): Ceph RBD takes care of it before individual chunks of an RBD
volume are passed to RADOS as objects and get written into the file
system as files by an OSD process.

The reason Fuel documentation recommends to disable QCOW2 format for
images is that RBD does not support QCOW2 disks directly, so Nova and
Cinder have to convert a QCOW2 image into RAW format before passing
it to QEMU's RBD driver. This means that you end up downloading the
QCOW2 image from Ceph to a nova-compute node (first full copy),
converting it (second full copy), and uploading the resultant RAW
image back to Ceph (third full copy) just to launch a VM or create a
volume from an image.

On Thu, May 28, 2015 at 8:33 AM, David Medberry openstack@medberry.net wrote:
yep. It's at the CEPH level (not the XFS level.)

On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins steve.cousins@maine.edu
wrote:

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry openstack@medberry.net
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure
how that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.flagg@gmail.com
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and qcow2
isn't recommended? Are there other factors to take into consideration that
I'm missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry openstack@medberry.net
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry
openstack@medberry.net wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete copy
instead of a zero byte copy as it cannot do its own copy-on-write tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Dmitry Borodaenko


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Even though we're using Ceph as a backend, we still use qcow2 images as our
golden images, since we still have a significant (maybe majority) number of
users using true ephemeral disks. It would be nice if glance was clever
enough to convert where appropriate.

Warren

Warren

On Thu, May 28, 2015 at 3:21 PM, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

I've experienced the opposite problem though. Downloading raw images and
uploading them to the cloud is very slow. Doing it through qcow2 allows
them to be compressed over the slow links. Ideally, the Ceph driver would
take a qcow2 and convert it to raw on glance ingest rather then at boot.

Thanks,
Kevin


From: Dmitry Borodaenko [dborodaenko@mirantis.com]
Sent: Thursday, May 28, 2015 12:10 PM
To: David Medberry
Cc: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] what is the different in use Qcow2 or
Raw in Ceph

David is right, Ceph implements volume snapshotting at the RBD level,
not even RADOS level: whole 2 levels of abstraction above file system.
It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
VFAT): Ceph RBD takes care of it before individual chunks of an RBD
volume are passed to RADOS as objects and get written into the file
system as files by an OSD process.

The reason Fuel documentation recommends to disable QCOW2 format for
images is that RBD does not support QCOW2 disks directly, so Nova and
Cinder have to convert a QCOW2 image into RAW format before passing
it to QEMU's RBD driver. This means that you end up downloading the
QCOW2 image from Ceph to a nova-compute node (first full copy),
converting it (second full copy), and uploading the resultant RAW
image back to Ceph (third full copy) just to launch a VM or create a
volume from an image.

On Thu, May 28, 2015 at 8:33 AM, David Medberry openstack@medberry.net
wrote:

yep. It's at the CEPH level (not the XFS level.)

On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins <
steve.cousins@maine.edu>
wrote:

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry <
openstack@medberry.net>
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure
how that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg <fostro.flagg@gmail.com

wrote:

I'm also curious about this. Here are some other pieces of
information
relevant to the discussion. Maybe someone here can clear this up for
me as
well. The documentation for Fuel 6.0, not sure what they changed for
6.1,
[1] states that when using Ceph one should disable qcow2 so that
images are
stored in raw format. This is due to the fact that Ceph includes its
own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system,
but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality.
Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw
images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and
qcow2
isn't recommended? Are there other factors to take into
consideration that
I'm missing?

[1]

https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2

[2]

http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/

[3]

https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry <
openstack@medberry.net>
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry
openstack@medberry.net wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete
copy
instead of a zero byte copy as it cannot do its own copy-on-write
tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of
the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.chen@gmail.com
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as
cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Dmitry Borodaenko


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded May 28, 2015 by Warren_Wang (680 points)   1
0 votes

We use ceph for glance and cinder, and we have some nodes that lack sufficient disk space, so they have "libvirtimagestype=rbd"
The majority of the "Public" images are raw format, which is pretty awesome because those huge windows images (20 - 80 GB Raw) spawn in 14s.
When a user uploads a qcow2 image, and launches it on a host that has rbd backend for ephemeral, nova.conf has "force_raw = true", so that they can run.
This adds a little bit of overhead, because nova has to download the qcow2 image (fast) from glance, then convert to raw (kinda fast) and write it to rbd (can be a bit slow)

It works, but long term, I've recommended that we eliminate the nodes that lack disk space, so we will switch to qcow2 by default, and I'm sure we'll eventually switch to swift for glance.

On May 28, 2015, at 12:34 PM, Warren Wang warren@wangspeed.com wrote:

Even though we're using Ceph as a backend, we still use qcow2 images as our golden images, since we still have a significant (maybe majority) number of users using true ephemeral disks. It would be nice if glance was clever enough to convert where appropriate.

Warren

Warren

On Thu, May 28, 2015 at 3:21 PM, Fox, Kevin M <Kevin.Fox@pnnl.gov Kevin.Fox@pnnl.gov> wrote:
I've experienced the opposite problem though. Downloading raw images and uploading them to the cloud is very slow. Doing it through qcow2 allows them to be compressed over the slow links. Ideally, the Ceph driver would take a qcow2 and convert it to raw on glance ingest rather then at boot.

Thanks,
Kevin


From: Dmitry Borodaenko [dborodaenko@mirantis.com dborodaenko@mirantis.com]
Sent: Thursday, May 28, 2015 12:10 PM
To: David Medberry
Cc: openstack-operators@lists.openstack.org openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] what is the different in use Qcow2 or Raw in Ceph

David is right, Ceph implements volume snapshotting at the RBD level,
not even RADOS level: whole 2 levels of abstraction above file system.
It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
VFAT): Ceph RBD takes care of it before individual chunks of an RBD
volume are passed to RADOS as objects and get written into the file
system as files by an OSD process.

The reason Fuel documentation recommends to disable QCOW2 format for
images is that RBD does not support QCOW2 disks directly, so Nova and
Cinder have to convert a QCOW2 image into RAW format before passing
it to QEMU's RBD driver. This means that you end up downloading the
QCOW2 image from Ceph to a nova-compute node (first full copy),
converting it (second full copy), and uploading the resultant RAW
image back to Ceph (third full copy) just to launch a VM or create a
volume from an image.

On Thu, May 28, 2015 at 8:33 AM, David Medberry <openstack@medberry.net openstack@medberry.net> wrote:

yep. It's at the CEPH level (not the XFS level.)

On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins <steve.cousins@maine.edu steve.cousins@maine.edu>
wrote:

Hi David,

So Ceph will use Copy-on-write even with XFS?

Thanks,

Steve

On Thu, May 28, 2015 at 10:36 AM, David Medberry <openstack@medberry.net openstack@medberry.net>
wrote:

This isn't remotely related to btrfs. It works fine with XFS. Not sure
how that works in Fuel, never used it.

On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg <fostro.flagg@gmail.com fostro.flagg@gmail.com>
wrote:

I'm also curious about this. Here are some other pieces of information
relevant to the discussion. Maybe someone here can clear this up for me as
well. The documentation for Fuel 6.0, not sure what they changed for 6.1,
[1] states that when using Ceph one should disable qcow2 so that images are
stored in raw format. This is due to the fact that Ceph includes its own
mechanisms for copy-on-write and snapshots. According to the Ceph
documentation [2], this is true only when using a BTRFS file system, but in
Fuel 6.0 Ceph uses XFS which doesn't provide this functionality. Also, [2]
recommends not using BTRFS for production as it isn't considered fully
mature. In addition, Fuel 6.0 [3] states that OpenStack with raw images
doesn't support snapshotting.

Given this, why does Fuel suggest not using qcow2 with Ceph? How can
Ceph be useful if snapshotting isn't an option with raw images and qcow2
isn't recommended? Are there other factors to take into consideration that
I'm missing?

[1]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2 https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
[2]
http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
[3]
https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

Thanks,

Forrest

On Thu, May 28, 2015 at 8:02 AM, David Medberry <openstack@medberry.net openstack@medberry.net>
wrote:

and better explained here:
http://ceph.com/docs/master/rbd/qemu-rbd/

On Thu, May 28, 2015 at 6:02 AM, David Medberry
<openstack@medberry.net openstack@medberry.net> wrote:

The primary difference is the ability for CEPH to make zero byte
copies. When you use qcow2, ceph must actually create a complete copy
instead of a zero byte copy as it cannot do its own copy-on-write tricks
with a qcow2 image.

So, yes, it will work fine with qcow2 images but it won't be as
performant as it is with RAW. Also, it will actually use more of the native
underlying storage.

This is also shown as an Important Note in the CEPH docs:
http://ceph.com/docs/master/rbd/rbd-openstack/

On Thu, May 28, 2015 at 4:12 AM, Shake Chen <shake.chen@gmail.com shake.chen@gmail.com>
wrote:

Hi

Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
nova and glance backend.

In Fuel document suggest if use ceph, suggest use RAW format image.

but if I upload qcow2 image, seem working well.

what is the different use qcow2 and RAW in Ceph?

--
Shake Chen


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--


Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Dmitry Borodaenko


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

responded May 28, 2015 by Abel_Lopez (4,820 points)   1 3 5
...