settingsLogin | Registersettings

[Openstack] Cinder error with Ceph backend, ImageCopyFailure: Failed to copy image to volume

0 votes

Hello, I have access to a running Ceph Storage Cluster. I am able to create
volumes with cinder, but not VMs (boot from image, creates new volume) Here
cinder-volume.log Anybody can help me? Thanks in advance. 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in dispatchandreply 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher incoming.message)) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher return self.
dodispatch(endpoint, method,
ctxt, args) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do
dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt,
newargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in
wrapper 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
return f(*args, **kwargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in
create
volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher runflow() 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in
runflow 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
flowengine.run() 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action
engine/engine.py",
line 89, in run 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher for state in self.runiter(): 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py",
line 137, in run
iter 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher
misc.Failure.reraiseifany(failures.values()) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in
reraiseifany 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher failures[0].reraise() 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in
reraise 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
six.reraise(*self._exc_info) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py",
line 34, in _execute_task 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = task.execute(
arguments) 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 638, in execute 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher **volume_spec) 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 590, in _create_from_image 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher image_id, image_location, image_service)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 504, in _copy_image_to_volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
ImageCopyFailure: Failed to copy image to volume: Image
94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked May 11, 2015 in openstack by Guillermo_Alvarado (740 points)   3 8

4 Responses

0 votes

I’ve been playing with Ceph quite a bit the last couple weeks and can probably help you out. The problem on looks to be:

Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.

Are you able to create any empty volumes successfully? If so, and assuming you followed the steps from http://docs.ceph.com/docs/master/rbd/rbd-openstack/, try this from your compute host:

rbd --user cinder --pool volumes ls

Also try that from the node where the cinder volume service is running. If you get an error trying that from either host, retrace the steps on the document above. If it works fine, then the issue is likely related to the "libvirtd secrets” section and making sure nova and libvirt are set up.

Let us know what you see and I’ll see if I can help.

Mike

Mike Smith
Principal Engineer, Website Systems
Overstock.com

On May 11, 2015, at 4:49 PM, Guillermo Alvarado guillermoalvarado89@gmail.com wrote:

Hello, I have access to a running Ceph Storage Cluster. I am able to create volumes with cinder, but not VMs (boot from image, creates new volume) Here cinder-volume.log Anybody can help me? Thanks in advance. 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in dispatchandreply 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher return self.dodispatch(endpoint, method, ctxt, args) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _dodispatch 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, newargs) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in createvolume 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher runflow() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in runflow 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher flowengine.run() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py", line 89, in run 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher for state in self.runiter(): 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py", line 137, in runiter 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher misc.Failure.reraiseifany(failures.values()) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in reraiseifany 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher failures[0].reraise() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in reraise 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher six.reraise(*self._exc_info) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 34, in _execute_task 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher result = task.execute(arguments) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 638, in execute 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher **volume_spec) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 590, in _create_from_image 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher image_id, image_location, image_service) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 504, in _copy_image_to_volume 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 11, 2015 by Mike_Smith (2,280 points)   1 4
0 votes

Hello, thanks Mike. Here the output:

Controller:

root@ctrl-net01:/var/lib# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Compute:
root@comp01:/etc/ceph# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Storage node(Ceph):

root@stor01:/etc/ceph# rbd --user cinder --pool volumes ls
2015-05-11 18:14:51.469680 7fe859185840 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2015-05-11 18:14:51.469686 7fe859185840 0 librados: client.cinder
initialization error (2) No such file or directory
rbd: couldn't connect to the cluster!

Also I recreated all the steps above but without glance configurations
because I do not want use RBd or ceph to my glance implementation.

Thanks in advance.

El lun., 11 de may. de 2015 a la(s) 6:06 p. m., Mike Smith <
mismith@overstock.com> escribió:

I’ve been playing with Ceph quite a bit the last couple weeks and can
probably help you out. The problem on looks to be:

Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.

Are you able to create any empty volumes successfully? If so, and
assuming you followed the steps from
http://docs.ceph.com/docs/master/rbd/rbd-openstack/, try this from your
compute host:

rbd --user cinder --pool volumes ls

Also try that from the node where the cinder volume service is running.
If you get an error trying that from either host, retrace the steps on
the document above. If it works fine, then the issue is likely related to
the "libvirtd secrets” section and making sure nova and libvirt are set up.

Let us know what you see and I’ll see if I can help.

Mike

Mike Smith
Principal Engineer, Website Systems
Overstock.com

On May 11, 2015, at 4:49 PM, Guillermo Alvarado <
guillermoalvarado89@gmail.com> wrote:

Hello, I have access to a running Ceph Storage Cluster. I am able to
create volumes with cinder, but not VMs (boot from image, creates new
volume) Here cinder-volume.log Anybody can help me? Thanks in advance.
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in dispatchandreply 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher incoming.message)) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher return self.
dodispatch(endpoint, method,
ctxt, args) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do
dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt,
newargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in
wrapper 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
return f(*args, **kwargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in
create
volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher runflow() 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in
runflow 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
flowengine.run() 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action
engine/engine.py",
line 89, in run 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher for state in self.runiter(): 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py",
line 137, in run
iter 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher
misc.Failure.reraiseifany(failures.values()) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in
reraiseifany 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher failures[0].reraise() 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in
reraise 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
six.reraise(*self._exc_info) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py",
line 34, in _execute_task 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = task.execute(
arguments) 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 638, in execute 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher **volume_spec) 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 590, in _create_from_image 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher image_id, image_location, image_service)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 504, in _copy_image_to_volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
ImageCopyFailure: Failed to copy image to volume: Image
94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

CONFIDENTIALITY NOTICE: This message is intended only for the use and
review of the individual or entity to which it is addressed and may contain
information that is privileged and confidential. If the reader of this
message is not the intended recipient, or the employee or agent responsible
for delivering the message solely to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this
communication in error, please notify sender immediately by telephone or
return email. Thank you.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 11, 2015 by Guillermo_Alvarado (740 points)   3 8
0 votes

OK, so it looks like your controller and compute nodes have the proper keyrings in place and are allowed to view the RBDs.

(On the storage node you probably can run without the “—user cinder” flag and it will work as long you have the correct /etc/ceph/ceph.client.admin.keyring in place)

What are you using for your glance backend? From the error I can’t tell if it’s failing to create the (ceph-based) volume image or failing to read the glance image.

Also, do you get a valid response back from “virsh secret-list” on the compute host, showing you the secret token you defined for use with cephx authentication?

Mike

On May 11, 2015, at 5:19 PM, Guillermo Alvarado guillermoalvarado89@gmail.com wrote:

Hello, thanks Mike. Here the output:

Controller:

root@ctrl-net01:/var/lib# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Compute:
root@comp01:/etc/ceph# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Storage node(Ceph):

root@stor01:/etc/ceph# rbd --user cinder --pool volumes ls
2015-05-11 18:14:51.469680 7fe859185840 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2015-05-11 18:14:51.469686 7fe859185840 0 librados: client.cinder initialization error (2) No such file or directory
rbd: couldn't connect to the cluster!

Also I recreated all the steps above but without glance configurations because I do not want use RBd or ceph to my glance implementation.

Thanks in advance.

El lun., 11 de may. de 2015 a la(s) 6:06 p. m., Mike Smith mismith@overstock.com escribió:
I’ve been playing with Ceph quite a bit the last couple weeks and can probably help you out. The problem on looks to be:

Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.

Are you able to create any empty volumes successfully? If so, and assuming you followed the steps from http://docs.ceph.com/docs/master/rbd/rbd-openstack/, try this from your compute host:

rbd --user cinder --pool volumes ls

Also try that from the node where the cinder volume service is running. If you get an error trying that from either host, retrace the steps on the document above. If it works fine, then the issue is likely related to the "libvirtd secrets” section and making sure nova and libvirt are set up.

Let us know what you see and I’ll see if I can help.

Mike

Mike Smith
Principal Engineer, Website Systems
Overstock.com

On May 11, 2015, at 4:49 PM, Guillermo Alvarado guillermoalvarado89@gmail.com wrote:

Hello, I have access to a running Ceph Storage Cluster. I am able to create volumes with cinder, but not VMs (boot from image, creates new volume) Here cinder-volume.log Anybody can help me? Thanks in advance. 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in dispatchandreply 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher return self.dodispatch(endpoint, method, ctxt, args) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _dodispatch 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, newargs) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in createvolume 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher runflow() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in runflow 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher flowengine.run() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py", line 89, in run 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher for state in self.runiter(): 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py", line 137, in runiter 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher misc.Failure.reraiseifany(failures.values()) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in reraiseifany 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher failures[0].reraise() 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in reraise 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher six.reraise(*self._exc_info) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 34, in _execute_task 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher result = task.execute(arguments) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 638, in execute 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher **volume_spec) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 590, in _create_from_image 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher image_id, image_location, image_service) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 504, in _copy_image_to_volume 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex) 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 12, 2015 by Mike_Smith (2,280 points)   1 4
0 votes

Hello Mike,

Yeah I can run the command withou "--user cinder" and works fine. Also I do
"virsh secret-list" on the compute host and works fine.

I am working with a HA installation, HAproxy, Galera and RabbitMQ in
cluster mode, can be related to that?

Glance is storing local in the controller.

Thanks for your help!

El mar., 12 de may. de 2015 a la(s) 8:52 a. m., Mike Smith <
mismith@overstock.com> escribió:

OK, so it looks like your controller and compute nodes have the proper
keyrings in place and are allowed to view the RBDs.

(On the storage node you probably can run without the “—user cinder”
flag and it will work as long you have the
correct /etc/ceph/ceph.client.admin.keyring in place)

What are you using for your glance backend? From the error I can’t
tell if it’s failing to create the (ceph-based) volume image or failing to
read the glance image.

Also, do you get a valid response back from “virsh secret-list” on the
compute host, showing you the secret token you defined for use with cephx
authentication?

Mike

On May 11, 2015, at 5:19 PM, Guillermo Alvarado <
guillermoalvarado89@gmail.com> wrote:

Hello, thanks Mike. Here the output:

Controller:

root@ctrl-net01:/var/lib# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Compute:
root@comp01:/etc/ceph# rbd --user cinder --pool volumes ls
testa1
volume-2b20fca2-752b-4c93-9dd1-600c96b2a6c4
volume-526df27f-f8a5-42c0-96f1-1976ff1cc1a9
volume-a15e66ee-5d9a-4032-a888-74a173584fd0
volume-b49d05bc-07e5-4104-9adf-4d7e2daf7839

Storage node(Ceph):

root@stor01:/etc/ceph# rbd --user cinder --pool volumes ls
2015-05-11 18:14:51.469680 7fe859185840 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2015-05-11 18:14:51.469686 7fe859185840 0 librados: client.cinder
initialization error (2) No such file or directory
rbd: couldn't connect to the cluster!

Also I recreated all the steps above but without glance configurations
because I do not want use RBd or ceph to my glance implementation.

Thanks in advance.

El lun., 11 de may. de 2015 a la(s) 6:06 p. m., Mike Smith <
mismith@overstock.com> escribió:

I’ve been playing with Ceph quite a bit the last couple weeks and can
probably help you out. The problem on looks to be:

Image 94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.

Are you able to create any empty volumes successfully? If so, and
assuming you followed the steps from
http://docs.ceph.com/docs/master/rbd/rbd-openstack/, try this from your
compute host:

rbd --user cinder --pool volumes ls

Also try that from the node where the cinder volume service is running.
If you get an error trying that from either host, retrace the steps on
the document above. If it works fine, then the issue is likely related to
the "libvirtd secrets” section and making sure nova and libvirt are set up.

Let us know what you see and I’ll see if I can help.

Mike

Mike Smith
Principal Engineer, Website Systems
Overstock.com

    On May 11, 2015, at 4:49 PM, Guillermo Alvarado <

guillermoalvarado89@gmail.com> wrote:

    Hello, I have access to a running Ceph Storage Cluster. I am

able to create volumes with cinder, but not VMs (boot from image, creates
new volume) Here cinder-volume.log Anybody can help me? Thanks in advance.
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in dispatchandreply 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher incoming.message)) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher return self.
dodispatch(endpoint, method,
ctxt, args) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do
dispatch 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt,
newargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in
wrapper 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
return f(*args, **kwargs) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in
create
volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher runflow() 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in
runflow 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
flowengine.run() 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action
engine/engine.py",
line 89, in run 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher for state in self.runiter(): 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/actionengine/engine.py",
line 137, in run
iter 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher
misc.Failure.reraiseifany(failures.values()) 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in
reraiseifany 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher failures[0].reraise() 2015-05-11 17:39:27.879
58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in
reraise 2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
six.reraise(*self._exc_info) 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py",
line 34, in _execute_task 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher result = task.execute(
arguments) 2015-05-11
17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 638, in execute 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher **volume_spec) 2015-05-11 17:39:27.879 58175
TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 590, in _create_from_image 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher image_id, image_location, image_service)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py",
line 504, in _copy_image_to_volume 2015-05-11 17:39:27.879 58175 TRACE
oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex)
2015-05-11 17:39:27.879 58175 TRACE oslo.messaging.rpc.dispatcher
ImageCopyFailure: Failed to copy image to volume: Image
94dc5715-7ce6-410b-90a9-d9a2f9b2e1e5 could not be found.

   _______________________________________________

Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


CONFIDENTIALITY NOTICE: This message is intended only for the use and
review of the individual or entity to which it is addressed and may contain
information that is privileged and confidential. If the reader of this
message is not the intended recipient, or the employee or agent responsible
for delivering the message solely to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this
communication in error, please notify sender immediately by telephone or
return email. Thank you.


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 12, 2015 by Guillermo_Alvarado (740 points)   3 8
...