settingsLogin | Registersettings

[Openstack] Urgent: How to recover a Controller?

0 votes

Hello, everyone!
My question is: My question is: How to recover a Controller node from
an accidental deletion?

Last weekend, my company programmed an electricity cut to make some

tests. to avoid harms in our production environment, so we decided to turn
it off using the procedure described here:
https://docs.openstack.org/developer/fuel-docs/userdocs/
fuel-user-guide/maintain-environment/shutdown-env.html
and to turn it on we follows:
https://docs.openstack.org/developer/fuel-docs/userdocs/
fuel-user-guide/maintain-environment/start-env.html#start-env

We started the node "Fuel Master" without problems. However, when we
started the controller 1, the problems began:
because of an old "Cluster Manager" installation that shouldn't be started,
the Controller1 was overwritten with another operating system.
Now, I need to recover the controller1 maybe using the controller2.

Please, can you tell me the procedure to do that?
Thanks in advance.

Here is my environment:

1 x Master (Fuel 9.0)
2 x Controller
1 x Storage (Cinder)
1 x telemetry-MongoDB (Celiometer)
13 x Compute

OpenStack Release: Mitaka on Ubuntu 14.04
Compute: KVM
Network: * Neutron with tunneling segmentation
*Storage Backends:
Cinder LVM over iSCSI for volumes, qcow images

Nodes features:
CPU 8 x 2.53 GHz
Disks 1 drive, 233.8 GB total
Memory 6 x 4.0 GB, 24.0 GB total
System Supermicro X8DA3

*Interfaces*     2 x 1.0 Gbps
                                        enp3s0      Admin (PXE)
                                        enp4s0      Public, Storage

(vlan:102), Management (vlan:101), Private(vlan: 103)

Switches:
DELL 6248 (enp3s0)
HP 5400zl (enp4s0)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Apr 20, 2017 in openstack by smachara.openstack_a (160 points)  

2 Responses

0 votes

Hi, everyone
I haven't a solution yet. I need your help, please
Thanks.

On Thu, Apr 13, 2017 at 5:10 PM, Samer MACHARA <smachara.openstack@gmail.com
wrote:

Hi Christian
thanks for answered me.

I tried to deploy a new Controller but I get the message:
* Deployment cannot be started due to invalid environment
configuration. Please review and address the warnings below before
proceeding. 1 node is offline*
So I did the provisioning only and them the deployment only, But I get an
error during the process(I lost the error once I delete the node), so I
Tried to delete the new controller and the offline node.

I use these commands to delete the nodes:

fuel node --node-id 106 --delete-from-db --force
fuel node --node-id 106 --delete --force

fuel node --node-id 89 --delete --force
fuel node --node-id 89 --delete-from-db --force

then I tried to deploy with
fuel deploy-changes --env 7

but nothing happens, Now, both nodes have the following status:

[root@fuel ~]# fuel node list
id | status | name | cluster | ip |
mac | roles | pendingroles | online | groupid
----+----------+---------------------+---------+-----------
-+-------------------+------------+---------------+--------+---------
89 | removing | Controller2 (5f:5a) | 7 | 10.20.0.7 |
00:30:48:bf:5f:5a | controller | | | 7
106 | removing | Controller3 (5d:50) | 7 | 10.20.0.9 |
00:30:48:bf:5d:50 | | controller | 1 | 7

and in the Fuel UI

Controller2 (5f:5a) controller Pending Deletion
Controller3 (5d:50) controller Removing

If I understood, both nodes will be deleted in the next deployment, but I
have nothing to deploy until both nodes are free. the "Deploy Changes"
button is disabled.

Thanks
Samer

On Wed, Apr 12, 2017 at 4:34 PM, Christian Berendt <
berendt@betacloud-solutions.de> wrote:

On 12 Apr 2017, at 10:39, Samer MACHARA smachara.openstack@gmail.com
wrote:

We started the node "Fuel Master" without problems. However, when we
started the controller 1, the problems began:
because of an old "Cluster Manager" installation that shouldn't be
started, the Controller1 was overwritten with another operating system.
Now, I need to recover the controller1 maybe using the controller2.

In theory it should be possible to make a new rollout/deployment of the
first controller. Have you already tried this?

Christian.

--
Christian Berendt
Chief Executive Officer (CEO)

Mail: berendt@betacloud-solutions.de
Web: https://www.betacloud-solutions.de

Betacloud Solutions GmbH
Teckstrasse 62 / 70190 Stuttgart / Deutschland

Geschäftsführer: Christian Berendt
Unternehmenssitz: Stuttgart
Amtsgericht: Stuttgart, HRB 756139


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Apr 20, 2017 by smachara.openstack_a (160 points)  
0 votes

fuel task list|grep -e 'pending|run'
delete those tasks forcefully and then do delete-from-db --force again .
add new node assign role controller.Deploy changes again

as far as i know --delete-from-db --force doesnt need any deploy
changes .It just disappeares from db and from "fuel node" list .

On Thu, Apr 20, 2017 at 1:45 PM, Samer MACHARA
smachara.openstack@gmail.com wrote:
Hi, everyone
I haven't a solution yet. I need your help, please
Thanks.

On Thu, Apr 13, 2017 at 5:10 PM, Samer MACHARA
smachara.openstack@gmail.com wrote:

Hi Christian
thanks for answered me.

I tried to deploy a new Controller but I get the message:
Deployment cannot be started due to invalid environment
configuration. Please review and address the warnings below before
proceeding. 1 node is offline
So I did the provisioning only and them the deployment only, But I get an
error during the process(I lost the error once I delete the node), so I
Tried to delete the new controller and the offline node.

I use these commands to delete the nodes:

fuel node --node-id 106 --delete-from-db --force
fuel node --node-id 106 --delete --force

fuel node --node-id 89 --delete --force
fuel node --node-id 89 --delete-from-db --force

then I tried to deploy with
fuel deploy-changes --env 7

but nothing happens, Now, both nodes have the following status:

[root@fuel ~]# fuel node list
id | status | name | cluster | ip | mac
| roles | pendingroles | online | groupid

----+----------+---------------------+---------+------------+-------------------+------------+---------------+--------+---------
89 | removing | Controller2 (5f:5a) | 7 | 10.20.0.7 |
00:30:48:bf:5f:5a | controller | | | 7
106 | removing | Controller3 (5d:50) | 7 | 10.20.0.9 |
00:30:48:bf:5d:50 | | controller | 1 | 7

and in the Fuel UI

Controller2 (5f:5a) controller Pending Deletion
Controller3 (5d:50) controller Removing

If I understood, both nodes will be deleted in the next deployment, but I
have nothing to deploy until both nodes are free. the "Deploy Changes"
button is disabled.

Thanks
Samer

On Wed, Apr 12, 2017 at 4:34 PM, Christian Berendt
berendt@betacloud-solutions.de wrote:

On 12 Apr 2017, at 10:39, Samer MACHARA smachara.openstack@gmail.com
wrote:

We started the node "Fuel Master" without problems. However, when we
started the controller 1, the problems began:
because of an old "Cluster Manager" installation that shouldn't be
started, the Controller1 was overwritten with another operating system.
Now, I need to recover the controller1 maybe using the controller2.

In theory it should be possible to make a new rollout/deployment of the
first controller. Have you already tried this?

Christian.

--
Christian Berendt
Chief Executive Officer (CEO)

Mail: berendt@betacloud-solutions.de
Web: https://www.betacloud-solutions.de

Betacloud Solutions GmbH
Teckstrasse 62 / 70190 Stuttgart / Deutschland

Geschäftsführer: Christian Berendt
Unternehmenssitz: Stuttgart
Amtsgericht: Stuttgart, HRB 756139


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Apr 20, 2017 by mad_Engineer (2,000 points)   1 9 15
...