settingsLogin | Registersettings

[Openstack-operators] Cannot delete a cinder volume; alternative manual procedure ?

0 votes

Hi,
I've a cinder volume which is permanently in "deleting" state. I cannot
retrace the full history of actions that brought to this scenario. What
I cat reporto is that

  1. Cinder is backed by GlusterFS mounted with the glusterfuse,
  2. a "cinder delete" (or "cinder force-delete" as admin) doesn't produce
    any effect,
  3. In the api.log, scheduler.log and volume.log I do not see any useful
    information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 INFO eventlet.wsgi.server
[req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25
d7b3d4f7d20444adb7fda140553c25bf 3beba6dd3f2648378263bc04d9c205fa - - -]
192.135.16.31,192.168.60.21 - - [05/Jun/2015 14:45:20] "DELETE
/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152
HTTP/1.1" 400 338 0.053790

(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume cannot
be deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant cinder's
tables) which gracefully cleans things up (which is not a trivial
"delete from volume where id='...'") exists.

Thanks,

 Alvise


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Jun 9, 2015 in openstack-operators by Alvise_Dorigo (1,380 points)   1 7 9

5 Responses

0 votes

Hi,

I also had once an issues like this. The deletion of the volume was
successfull but the DB was not updated after volume was deleted.
I solved it by manually creating the volume again (ceph in our case) the
size doesn't matter. After I could delete with cinder commands.

Hope this helps

Benedikt

On 09.06.2015 15:37, Alvise Dorigo wrote:
Hi,
I've a cinder volume which is permanently in "deleting" state. I cannot
retrace the full history of actions that brought to this scenario. What
I cat reporto is that

  1. Cinder is backed by GlusterFS mounted with the glusterfuse,
  2. a "cinder delete" (or "cinder force-delete" as admin) doesn't produce
    any effect,
  3. In the api.log, scheduler.log and volume.log I do not see any useful
    information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 INFO eventlet.wsgi.server
[req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25
d7b3d4f7d20444adb7fda140553c25bf 3beba6dd3f2648378263bc04d9c205fa - - -]
192.135.16.31,192.168.60.21 - - [05/Jun/2015 14:45:20] "DELETE
/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152
HTTP/1.1" 400 338 0.053790

(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume cannot
be deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant cinder's
tables) which gracefully cleans things up (which is not a trivial
"delete from volume where id='...'") exists.

Thanks,

Alvise


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 9, 2015 by Benedikt_Trefzer (160 points)   2
0 votes

We've used
https://github.com/snemetz/openstack-scripts/blob/master/cinder-volume-delete.txt
with apparent success to delete stuck volumes.

  • jlk

On Tue, Jun 9, 2015 at 6:37 AM, Alvise Dorigo alvise.dorigo@pd.infn.it
wrote:

Hi,
I've a cinder volume which is permanently in "deleting" state. I cannot
retrace the full history of actions that brought to this scenario. What I
cat reporto is that

  1. Cinder is backed by GlusterFS mounted with the glusterfuse,
  2. a "cinder delete" (or "cinder force-delete" as admin) doesn't produce
    any effect,
  3. In the api.log, scheduler.log and volume.log I do not see any useful
    information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 INFO eventlet.wsgi.server
[req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25 d7b3d4f7d20444adb7fda140553c25bf
3beba6dd3f2648378263bc04d9c205fa - - -] 192.135.16.31,192.168.60.21 - -
[05/Jun/2015 14:45:20] "DELETE
/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152
HTTP/1.1" 400 338 0.053790

(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume cannot be
deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant cinder's
tables) which gracefully cleans things up (which is not a trivial "delete
from volume where id='...'") exists.

Thanks,

Alvise


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 9, 2015 by jlk_at_bluebox.net (2,760 points)   2 3
0 votes

On 6/9/2015 11:28 AM, Jesse Keating wrote:
We've used
https://github.com/snemetz/openstack-scripts/blob/master/cinder-volume-delete.txt
with apparent success to delete stuck volumes.

  • jlk

On Tue, Jun 9, 2015 at 6:37 AM, Alvise Dorigo <alvise.dorigo@pd.infn.it
alvise.dorigo@pd.infn.it> wrote:

Hi,
I've a cinder volume which is permanently in "deleting" state. I
cannot retrace the full history of actions that brought to this
scenario. What I cat reporto is that

1. Cinder is backed by GlusterFS mounted with the glusterfuse,
2. a "cinder delete" (or "cinder force-delete" as admin) doesn't
produce any effect,
3. In the api.log, scheduler.log and volume.log I do not see any
useful information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 <tel:20.956%2017383>
INFO eventlet.wsgi.server [req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25
d7b3d4f7d20444adb7fda140553c25bf 3beba6dd3f2648378263bc04d9c205fa -
- -] 192.135.16.31,192.168.60.21 - - [05/Jun/2015 14:45:20] "DELETE
/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152
HTTP/1.1" 400 338 0.053790


(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume
cannot be deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant
cinder's tables) which gracefully cleans things up (which is not a
trivial "delete from volume where id='...'") exists.

Thanks,

     Alvise

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
<mailto:OpenStack-operators@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

FYI, there is an effort going on here [1] to add a force volume
attachment removal command to nova-manage. Operator input on that change
would be valuable.

[1] https://review.openstack.org/#/c/184537/

--

Thanks,

Matt Riedemann


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Oct 26, 2015 by Matt_Riedemann (48,320 points)   3 7 19
0 votes

I'm guessing that you are probably using multiple cinder-volume services to
manage the glusterfs volumes. If this is the case, and you didn't set
'host' in cinder.conf to a common name between the cinder-volume services
then you can end up in a case where you can't delete a volume because the
registered 'host' is offline or no longer around.

On Tue, Oct 27, 2015 at 12:43 AM Matt Riedemann mriedem@linux.vnet.ibm.com
wrote:

On 6/9/2015 11:28 AM, Jesse Keating wrote:

We've used

https://github.com/snemetz/openstack-scripts/blob/master/cinder-volume-delete.txt

with apparent success to delete stuck volumes.

  • jlk

On Tue, Jun 9, 2015 at 6:37 AM, Alvise Dorigo <alvise.dorigo@pd.infn.it
alvise.dorigo@pd.infn.it> wrote:

Hi,
I've a cinder volume which is permanently in "deleting" state. I
cannot retrace the full history of actions that brought to this
scenario. What I cat reporto is that

1. Cinder is backed by GlusterFS mounted with the glusterfuse,
2. a "cinder delete" (or "cinder force-delete" as admin) doesn't
produce any effect,
3. In the api.log, scheduler.log and volume.log I do not see any
useful information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 <tel:20.956%2017383>
INFO eventlet.wsgi.server [req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25
d7b3d4f7d20444adb7fda140553c25bf 3beba6dd3f2648378263bc04d9c205fa -
- -] 192.135.16.31,192.168.60.21 - - [05/Jun/2015 14:45:20] "DELETE

/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152

HTTP/1.1" 400 338 0.053790


(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume
cannot be deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant
cinder's tables) which gracefully cleans things up (which is not a
trivial "delete from volume where id='...'") exists.

Thanks,

     Alvise

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
<mailto:OpenStack-operators@lists.openstack.org>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

FYI, there is an effort going on here [1] to add a force volume
attachment removal command to nova-manage. Operator input on that change
would be valuable.

[1] https://review.openstack.org/#/c/184537/

--

Thanks,

Matt Riedemann


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--

--

Andrew Woodward

Mirantis

Fuel Community Ambassador

Ceph Community


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Oct 28, 2015 by Andrew_Woodward (6,960 points)   2 3 5
0 votes

Is this method, using a common name when using shared storage backend,
documented somewhere?

  • jlk

On Tue, Oct 27, 2015 at 7:41 PM, Andrew Woodward xarses@gmail.com wrote:

I'm guessing that you are probably using multiple cinder-volume services
to manage the glusterfs volumes. If this is the case, and you didn't set
'host' in cinder.conf to a common name between the cinder-volume services
then you can end up in a case where you can't delete a volume because the
registered 'host' is offline or no longer around.

On Tue, Oct 27, 2015 at 12:43 AM Matt Riedemann <
mriedem@linux.vnet.ibm.com> wrote:

On 6/9/2015 11:28 AM, Jesse Keating wrote:

We've used

https://github.com/snemetz/openstack-scripts/blob/master/cinder-volume-delete.txt

with apparent success to delete stuck volumes.

  • jlk

On Tue, Jun 9, 2015 at 6:37 AM, Alvise Dorigo <alvise.dorigo@pd.infn.it
alvise.dorigo@pd.infn.it> wrote:

Hi,
I've a cinder volume which is permanently in "deleting" state. I
cannot retrace the full history of actions that brought to this
scenario. What I cat reporto is that

1. Cinder is backed by GlusterFS mounted with the glusterfuse,
2. a "cinder delete" (or "cinder force-delete" as admin) doesn't
produce any effect,
3. In the api.log, scheduler.log and volume.log I do not see any
useful information but this:

api.log-20150607:2015-06-05 14:45:20.956 17383 <tel:20.956%2017383>
INFO eventlet.wsgi.server [req-bed01706-77ff-48bb-b12c-a0ccdcfd2e25
d7b3d4f7d20444adb7fda140553c25bf 3beba6dd3f2648378263bc04d9c205fa -
- -] 192.135.16.31,192.168.60.21 - - [05/Jun/2015 14:45:20] "DELETE

/v1/3beba6dd3f2648378263bc04d9c205fa/volumes/b056b016-8337-4e64-9069-6c84ab242152

HTTP/1.1" 400 338 0.053790


(the verbosity and debug are ON of course).

This is not a single event... unfortunately more that one volume
cannot be deleted and we're, so far, unable to understand why.

My question is if a manual procedure (operating on the relevant
cinder's tables) which gracefully cleans things up (which is not a
trivial "delete from volume where id='...'") exists.

Thanks,

     Alvise

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
<mailto:OpenStack-operators@lists.openstack.org>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

FYI, there is an effort going on here [1] to add a force volume
attachment removal command to nova-manage. Operator input on that change
would be valuable.

[1] https://review.openstack.org/#/c/184537/

--

Thanks,

Matt Riedemann


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--

--

Andrew Woodward

Mirantis

Fuel Community Ambassador

Ceph Community


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Nov 3, 2015 by jlk_at_bluebox.net (2,760 points)   2 3
...