settingsLogin | Registersettings

[Openstack] Compute Node shutdown how to prevent instance suspend

0 votes

Hello,

When we shut down a compute node the instances running on it get
suspended. This generates some difficulties with some applications like
RabbitMQ dont like to be suspended. Is there a way to change this
behavior so that the running instances gets killed or shutdown instead?

Thanks in advance.

Cheers,
Chris


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Nov 3, 2017 in openstack by Chris (1,320 points)   6 8

8 Responses

0 votes

Hi,

A better Q would be why do you shutdown a compute node to begin with?
I mean if you need you should do so in an orderly fashion basically
excavate instances
or shut instances down manually, put the compute node in maintenance mode.
On rebooting compute node remove it from maintenance mode, turn on
instances or migrate them back to this compute node should you need.
Od delete them if you wish.

There is this nova option:
resumeguestsstateonhost_boot=true

But it doesn't delete or shutdown instances but rather turns them on
automatically once compute host resumes.
which might also work for you, probably not just mentioning it any way.

I don't know of an option to stop/delete instance on compute node shutdown.

Another option check maybe you could shelve suspended instance and then
later delete them.

Shelving stops the instance and takes a snapshot of it. Then depending on
the value of the shelvedoffloadtime config option, the instance is
deleted from the hypervisor (0), never deleted (-1), or deleted after some
period of time (> 0). Note that it's just destroying the backing instance
on the hypervisor, the actual instance in the nova database is not deleted.
Then you can later unshelve the instance:

This might help, but do not if you mess with kvm without updating Nova you
might be left haning else where :)
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/

On Thu, Nov 2, 2017 at 9:03 AM, Chris contact@progbau.de wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This generates some difficulties with some applications like
RabbitMQ dont like to be suspended. Is there a way to change this behavior
so that the running instances gets killed or shutdown instead?

Thanks in advance.

Cheers,
Chris


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k

--

Tzach Shefi

Senior Quality Engineer, RHCSA

Red Hat

tshefi@redaht.com M: +972-54-4701080 IM: tshefi

responded Nov 2, 2017 by Tzach_Shefi (760 points)   1
0 votes

Hi!

When I need to reboot a compute node (because of some driver lock-up or such problems), I first stop nova-compute so that it does not report the shutoff state of VMs to the database and resumeguestsstateonhost_boot=true does actually start them. Then I press the power switch on the VMs using virsh shutdown, one by one, and only after that I reboot the node. This is a script I scrounged somewhere and modified so that it does not take too much time:

Tomas

root@cmp03:~# cat vm-shutdown

!/bin/bash

file: /usr/local/sbin/vm-shutdown

Description: shutdown active virtual machines

debug=1

fake=1

Get list of active virtual machines

vmList="`virsh list | (

while read vmID vmName vmStatus

do

  if [ -n "$vmName" -a "$vmName" != "Name" -a "$vmName" != "Domain-0" ]

  then

    [ -z "$vmList" ] && vmList="$vmName" || vmList="$vmList $vmName"

  fi

done

echo $vmList )`"

check there are some active VM's

if [ -n "$vmList" ]; then

  # Shutdown VM's with verification

  for vmName in $vmList

  do

    # send initial request

    [ -n "$debug" ] && echo -n "Attempting to shutdown $vmName "

    [ -z "$fake" ] && virsh shutdown $vmName

    # wait a limited time for the VM to be not running

    count=30

    while $( virsh list | grep $vmName >/dev/null ) && [ $count -gt 0 ]

    do

      sleep 1

      let count=count-1

      [ -n "$debug" ] && echo -n "."

    done

    # report current status

    ( virsh list | grep $vmName >/dev/null ) && echo " failed!" || echo "down."

    # if still running, destroy it

    if ( virsh list | grep $vmName >/dev/null )

    then

      [ -n "$debug" ] && echo -n "Attempting to destroy $vmName "

      [ -z "$fake" ] && virsh destroy $vmName

      # wait a limited time for the VM to be not running

      count=30

      while $( virsh list | grep $vmName >/dev/null ) && [ $count -gt 0 ]

      do

        sleep 1

        let count=count-1

        [ -n "$debug" ] && echo -n "."

      done

      # report current status

      ( virsh list | grep $vmName >/dev/null ) && echo " failed!" || echo "down."

    fi

  done

From: Tzach Shefi [mailto:tshefi@redhat.com]
Sent: Thursday, November 02, 2017 9:55 AM
To: Chris
Cc: openstack@lists.openstack.org
Subject: Re: [Openstack] Compute Node shutdown how to prevent instance suspend

Hi, 

A better Q would be why do you shutdown a compute node to begin with?

I mean if you need you should do so in an orderly fashion basically excavate instances

or shut instances down manually, put the compute node in maintenance mode. 

On rebooting compute node remove it from maintenance mode, turn on instances or migrate them back to this compute node should you need. 

Od delete them if you wish. 

There is this nova option:

resumeguestsstateonhost_boot=true

But it doesn't delete or shutdown instances but rather turns them on automatically once compute host resumes. 

which might also work for you, probably not just mentioning it any way.  

I don't know of an option to stop/delete instance on compute node shutdown. 

Another option check maybe you could shelve suspended instance and then later delete them. 

Shelving stops the instance and takes a snapshot of it. Then depending on the value of the shelvedoffloadtime config option, the instance is deleted from the hypervisor (0), never deleted (-1), or deleted after some period of time (> 0). Note that it's just destroying the backing instance on the hypervisor, the actual instance in the nova database is not deleted. Then you can later unshelve the instance:

This might help, but do not if you mess with kvm without updating Nova you might be left haning else where :)

https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/

On Thu, Nov 2, 2017 at 9:03 AM, Chris contact@progbau.de wrote:

Hello,

When we shut down a compute node the instances running on it get suspended. This generates some difficulties with some applications like RabbitMQ dont like to be suspended. Is there a way to change this behavior so that the running instances gets killed or shutdown instead?

Thanks in advance.

Cheers,
Chris


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

--

Tzach Shefi

Senior Quality Engineer, RHCSA

Red Hat

tshefi@redaht.com    M: +972-54-4701080     IM: tshefi


responded Nov 2, 2017 by vondra_at_homeatclou (780 points)  
0 votes

On 11/02/2017 01:03 AM, Chris wrote:
Hello,

When we shut down a compute node the instances running on it get suspended. This
generates some difficulties with some applications like RabbitMQ dont like to be
suspended. Is there a way to change this behavior so that the running instances
gets killed or shutdown instead?

This may be done by the libvirtd shutdown scripts rather than anything in nova.

As others have said, you should probably either shut down the VMs in an orderly
fashion or else cold/live migrate the instances off the compute node before
rebooting it.

responded Nov 2, 2017 by Chris_Friesen (20,420 points)   2 11 18
0 votes

If you are using CentOS  / Red Hat you can modify this file:

libvirt-guests:#ON_SHUTDOWN=suspend

action taken on host shutdown

- suspend   all running guests are suspended using virsh managedsave

- shutdown  all running guests are asked to shutdown. Please be

careful with

             this settings since there is no way to distinguish between a

             guest which is stuck or ignores shutdown requests and a guest

             which just needs a long time to shutdown. When setting

             ONSHUTDOWN=shutdown, you must also set SHUTDOWNTIMEOUT to a

             value suitable for your guests.

ON_SHUTDOWN=suspend

Hope this help.

Remo

On 11/2/17 11:08 AM, Chris Friesen wrote:
On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This
generates some difficulties with some applications like RabbitMQ dont
like to be
suspended. Is there a way to change this behavior so that the running
instances
gets killed or shutdown instead?

This may be done by the libvirtd shutdown scripts rather than anything
in nova.

As others have said, you should probably either shut down the VMs in
an orderly fashion or else cold/live migrate the instances off the
compute node before rebooting it.

Chris


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 2, 2017 by Remo_Mattei (10,500 points)   1 2 3
0 votes

sorry forgot to paste the file location
/etc/sysconfig/libvirt-guests

On 11/2/17 11:21 AM, Remo Mattei wrote:
If you are using CentOS  / Red Hat you can modify this file:

libvirt-guests:#ON_SHUTDOWN=suspend

action taken on host shutdown

- suspend   all running guests are suspended using virsh managedsave

- shutdown  all running guests are asked to shutdown. Please be

careful with

             this settings since there is no way to distinguish between a

             guest which is stuck or ignores shutdown requests and a guest

             which just needs a long time to shutdown. When setting

             ONSHUTDOWN=shutdown, you must also set SHUTDOWNTIMEOUT to a

             value suitable for your guests.

ON_SHUTDOWN=suspend

Hope this help.

Remo

On 11/2/17 11:08 AM, Chris Friesen wrote:

On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This
generates some difficulties with some applications like RabbitMQ dont
like to be
suspended. Is there a way to change this behavior so that the running
instances
gets killed or shutdown instead?
This may be done by the libvirtd shutdown scripts rather than anything
in nova.

As others have said, you should probably either shut down the VMs in
an orderly fashion or else cold/live migrate the instances off the
compute node before rebooting it.

Chris


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 2, 2017 by Remo_Mattei (10,500 points)   1 2 3
0 votes

There is this nova option:
resumeguestsstateonhost_boot=true

But it doesn't delete or shutdown instances but rather turns them on automatically once compute host resumes.
which might also work for you, probably not just mentioning it any way.

Note that you might want to check some things after a reboot before booting up all your instances.
The easiest way would be to set this value to true but to not start the nova-compute service on boot.
Now you can check whatever you need and start nova-compute when you know the server is in a good state.
Nova will now start the instances that where running when your compute node was shutdown.
The nice thing about this is that any instances that where shutdown/suspended by users will keep in a shutdown state.
If you do not do this you would need to create a custom script that saves which instances where running before the shutdown and then use that list to start only the correct instances.

Another thing to consider is if you want to auto-start instances after a compute node did an unexpected reboot.
IMHO I would rather have a dead instance then an instance running on flaky hard/software but it might depend a bit on your use case.

I don't know of an option to stop/delete instance on compute node shutdown.
As mentioned by some other people this is actually a libvirt setting not a nova setting.

Cheers,
Robert van Leeuwen


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 3, 2017 by Van_Leeuwen,_Robert (1,740 points)   3
0 votes

Hello Tzach,

Agree for a graceful shutdown there should be some steps ahead. Im
actually asking the question because the hardware we use has a high
failure rate on RAID controllers. The physical machine reacts with a
reboot, so its not controlable from our side. We dont want to delete the
instances at all, its just about the Instance shutdown/process kill.

Got a good answer already:
If you are using CentOS / Red Hat you can modify this file:
/etc/sysconfig/libvirt-guests
libvirt-guests:#ON_SHUTDOWN=suspend

Thanks for your answer!

Cheers Chris

On 2017-11-02 15:54, Tzach Shefi wrote:
Hi, 

A better Q would be why do you shutdown a compute node to begin with?
I mean if you need you should do so in an orderly fashion basically
excavate instances

or shut instances down manually, put the compute node in maintenance
mode. 
On rebooting compute node remove it from maintenance mode, turn on
instances or migrate them back to this compute node should you need. 
Od delete them if you wish. 

There is this nova option:
resumeguestsstateonhost_boot=true

But it doesn't delete or shutdown instances but rather turns them on
automatically once compute host resumes. 
which might also work for you, probably not just mentioning it any
way.  

I don't know of an option to stop/delete instance on compute node
shutdown. 

Another option check maybe you could shelve suspended instance and
then later delete them. 

Shelving stops the instance and takes a snapshot of it. Then depending
on the value of the shelvedoffloadtime config option, the instance
is deleted from the hypervisor (0), never deleted (-1), or deleted
after some period of time (> 0). Note that it's just destroying the
backing instance on the hypervisor, the actual instance in the nova
database is not deleted. Then you can later unshelve the instance:

This might help, but do not if you mess with kvm without updating Nova
you might be left haning else where :)
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/
[2]

 

On Thu, Nov 2, 2017 at 9:03 AM, Chris contact@progbau.de wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This generates some difficulties with some applications
like RabbitMQ dont like to be suspended. Is there a way to change
this behavior so that the running instances gets killed or shutdown
instead?

Thanks in advance.

Cheers,
Chris


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]

--

Tzach Shefi

Senior Quality Engineer, RHCSA

Red Hat

tshefi@redaht.com    M: +972-54-4701080 [3]     IM: tshefi

   [4]

Links:


[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[2]
https://ask.fedoraproject.org/en/question/8796/make-libvirt-to-shutdown-my-guests-not-suspend/
[3] tel:+972-54-4701080
[4] https://red.ht/sig


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 3, 2017 by Chris (1,320 points)   6 8
0 votes

Hello Remo,

Exactly what I was looking for! If I could give you a rating it would be
the highest :)

Cheers,
Chris

On 2017-11-03 1:27, Remo Mattei wrote:
sorry forgot to paste the file location
/etc/sysconfig/libvirt-guests

On 11/2/17 11:21 AM, Remo Mattei wrote:

If you are using CentOS  / Red Hat you can modify this file:

libvirt-guests:#ON_SHUTDOWN=suspend

action taken on host shutdown

- suspend   all running guests are suspended using virsh managedsave

- shutdown  all running guests are asked to shutdown. Please be

careful with

             this settings since there is no way to distinguish

between a

             guest which is stuck or ignores shutdown requests and a

guest

             which just needs a long time to shutdown. When setting

             ONSHUTDOWN=shutdown, you must also set SHUTDOWNTIMEOUT

to a

             value suitable for your guests.

ON_SHUTDOWN=suspend

Hope this help.

Remo

On 11/2/17 11:08 AM, Chris Friesen wrote:

On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get
suspended. This
generates some difficulties with some applications like RabbitMQ
dont
like to be
suspended. Is there a way to change this behavior so that the
running
instances
gets killed or shutdown instead?
This may be done by the libvirtd shutdown scripts rather than
anything
in nova.

As others have said, you should probably either shut down the VMs in
an orderly fashion or else cold/live migrate the instances off the
compute node before rebooting it.

Chris


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Nov 3, 2017 by Chris (1,320 points)   6 8
...