settingsLogin | Registersettings

[openstack-dev] [Heat] Tripleo holding on to old, bad data

0 votes

A coworker and I have both had trouble recovering from failed overcloud
deploys. I've wiped out whatever data I can, but, even with nothing in
the Heat Database, doing an

openstack overcloud deploy

seems to be looking for a specific Nova server by UUID:

heat resource-show 93afc25e-1ab2-4773-9949-6906e2f7c115 0

| resourcestatusreason | ResourceInError:
resources[0].resources.Controller: Went to status ERROR due
t│·········································································································
o "Message: No valid host was found. There are not enough hosts
available., Code: 500" |
│·········································································································
| resource_type | OS::TripleO::Controller

Inside the Nova log I see:

2016-06-24 21:05:06.973 15551 DEBUG nova.api.openstack.wsgi
[req-c8a5179c-2adf-45a6-b186-7d7b29cd8f39
bcd│·········································································································fefb36f44443ca9a8f3cfa445ab40
ec662f250a85453cb40054f3aff49b58 - - -] Returning 404 to user: Instance
8f9│·········································································································0c961-4609-4c9b-9d62-360a40f88eed
could not be found. call
/usr/lib/python2.7/site-packages/nova/api/│·········································································································
openstack/wsgi.py:1070

How can I get the undercloud back to a clean state?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 24, 2016 in openstack-dev by Adam_Young (19,940 points)   2 7 9
retagged Jan 26, 2017 by admin

4 Responses

0 votes

Assuming the stack is deleted and nova is showing no servers, you likely
have ironic nodes which are not in a state which can be scheduled.

Do an ironic node-list, you want Power State: Off, Provisioning State:
available, Maintenance: False

On 25/06/16 09:27, Adam Young wrote:
A coworker and I have both had trouble recovering from failed
overcloud deploys. I've wiped out whatever data I can, but, even with
nothing in the Heat Database, doing an

openstack overcloud deploy

seems to be looking for a specific Nova server by UUID:

heat resource-show 93afc25e-1ab2-4773-9949-6906e2f7c115 0

| resourcestatusreason | ResourceInError:
resources[0].resources.Controller: Went to status ERROR due
t│·········································································································
o "Message: No valid host was found. There are not enough hosts
available., Code: 500" |
│·········································································································
| resource_type | OS::TripleO::Controller

Inside the Nova log I see:

2016-06-24 21:05:06.973 15551 DEBUG nova.api.openstack.wsgi
[req-c8a5179c-2adf-45a6-b186-7d7b29cd8f39
bcd│·········································································································fefb36f44443ca9a8f3cfa445ab40
ec662f250a85453cb40054f3aff49b58 - - -] Returning 404 to user:
Instance
8f9│·········································································································0c961-4609-4c9b-9d62-360a40f88eed
could not be found. call
/usr/lib/python2.7/site-packages/nova/api/│·········································································································
openstack/wsgi.py:1070

How can I get the undercloud back to a clean state?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 26, 2016 by Steve_Baker (7,380 points)   1 3 6
0 votes

On 06/26/2016 07:00 PM, Steve Baker wrote:
Assuming the stack is deleted and nova is showing no servers, you
likely have ironic nodes which are not in a state which can be scheduled.

Do an ironic node-list, you want Power State: Off, Provisioning State:
available, Maintenance: False

Yes, we have that. First thing we checked. I assume "available" is the
most important part of that?

On 25/06/16 09:27, Adam Young wrote:

A coworker and I have both had trouble recovering from failed
overcloud deploys. I've wiped out whatever data I can, but, even
with nothing in the Heat Database, doing an

openstack overcloud deploy

seems to be looking for a specific Nova server by UUID:

heat resource-show 93afc25e-1ab2-4773-9949-6906e2f7c115 0

| resourcestatusreason | ResourceInError:
resources[0].resources.Controller: Went to status ERROR due
t│·········································································································
o "Message: No valid host was found. There are not enough hosts
available., Code: 500" |
│·········································································································
| resource_type | OS::TripleO::Controller

Inside the Nova log I see:

2016-06-24 21:05:06.973 15551 DEBUG nova.api.openstack.wsgi
[req-c8a5179c-2adf-45a6-b186-7d7b29cd8f39
bcd│·········································································································fefb36f44443ca9a8f3cfa445ab40
ec662f250a85453cb40054f3aff49b58 - - -] Returning 404 to user:
Instance
8f9│·········································································································0c961-4609-4c9b-9d62-360a40f88eed
could not be found. call
/usr/lib/python2.7/site-packages/nova/api/│·········································································································
openstack/wsgi.py:1070

How can I get the undercloud back to a clean state?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 27, 2016 by Adam_Young (19,940 points)   2 7 9
0 votes

Adam,

not only "available", Nova would also not schedule to Ironic nodes which
have maintenance==True regardless of their provisioning state.

Also, you might have orphaned Ironic nodes, when node is available, but
still has instance_uuid assigned without actual instance in Nova. These
AFAIK would also not be scheduled to. To fix it update the node resetting
this field

ironic node-update remove instance_uuid

Cheers,

Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com

On Tue, Jun 28, 2016 at 1:29 AM, Adam Young ayoung@redhat.com wrote:

On 06/26/2016 07:00 PM, Steve Baker wrote:

Assuming the stack is deleted and nova is showing no servers, you likely
have ironic nodes which are not in a state which can be scheduled.

Do an ironic node-list, you want Power State: Off, Provisioning State:
available, Maintenance: False

Yes, we have that. First thing we checked. I assume "available" is the
most important part of that?

On 25/06/16 09:27, Adam Young wrote:

A coworker and I have both had trouble recovering from failed overcloud
deploys. I've wiped out whatever data I can, but, even with nothing in the
Heat Database, doing an

openstack overcloud deploy

seems to be looking for a specific Nova server by UUID:

heat resource-show 93afc25e-1ab2-4773-9949-6906e2f7c115 0

| resourcestatusreason | ResourceInError:
resources[0].resources.Controller: Went to status ERROR due
t│·········································································································
o "Message: No valid host was found. There are not enough hosts
available., Code: 500" |
│·········································································································
| resource_type | OS::TripleO::Controller

Inside the Nova log I see:

2016-06-24 21:05:06.973 15551 DEBUG nova.api.openstack.wsgi
[req-c8a5179c-2adf-45a6-b186-7d7b29cd8f39
bcd│·········································································································fefb36f44443ca9a8f3cfa445ab40
ec662f250a85453cb40054f3aff49b58 - - -] Returning 404 to user: Instance
8f9│·········································································································0c961-4609-4c9b-9d62-360a40f88eed
could not be found. call
/usr/lib/python2.7/site-packages/nova/api/│·········································································································
openstack/wsgi.py:1070

How can I get the undercloud back to a clean state?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 28, 2016 by Pavlo_Shchelokovskyy (4,760 points)   4 5
0 votes

On 06/28/2016 02:58 AM, Pavlo Shchelokovskyy wrote:
Adam,

not only "available", Nova would also not schedule to Ironic nodes
which have maintenance==True regardless of their provisioning state.
That was not set.

Also, you might have orphaned Ironic nodes, when node is available,
but still has instance_uuid assigned without actual instance in Nova.
These AFAIK would also not be scheduled to. To fix it update the node
resetting this field

ironic node-update remove instance_uuid

Did that as well, but since the system has been rebuilt, it is hard to
confirm. If we do it again, I'll double check all these. Thanks

Cheers,

Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com

On Tue, Jun 28, 2016 at 1:29 AM, Adam Young <ayoung@redhat.com
ayoung@redhat.com> wrote:

On 06/26/2016 07:00 PM, Steve Baker wrote:

    Assuming the stack is deleted and nova is showing no servers,
    you likely have ironic nodes which are not in a state which
    can be scheduled.

    Do an ironic node-list, you want Power State: Off,
    Provisioning State: available, Maintenance: False


Yes, we have that.  First thing we checked.  I assume "available"
is the most important part of that?




    On 25/06/16 09:27, Adam Young wrote:

        A coworker and I have both had trouble recovering from
        failed overcloud deploys.  I've wiped out whatever data I
        can, but, even with nothing in the Heat Database, doing an

        openstack overcloud deploy

        seems to be looking for a specific Nova server by UUID:


        heat resource-show 93afc25e-1ab2-4773-9949-6906e2f7c115 0

        | resource_status_reason | ResourceInError:
        resources[0].resources.Controller: Went to status ERROR
        due
        t│·········································································································
        o "Message: No valid host was found. There are not enough
        hosts available., Code: 500" |
        │·········································································································
        | resource_type          | OS::TripleO::Controller


        Inside the Nova log I see:


        2016-06-24 21:05:06.973 15551 DEBUG
        nova.api.openstack.wsgi
        [req-c8a5179c-2adf-45a6-b186-7d7b29cd8f39
        bcd│·········································································································fefb36f44443ca9a8f3cfa445ab40
        ec662f250a85453cb40054f3aff49b58 - - -] Returning 404 to
        user: Instance
        8f9│·········································································································0c961-4609-4c9b-9d62-360a40f88eed
        could not be found. __call__
        /usr/lib/python2.7/site-packages/nova/api/│·········································································································
        openstack/wsgi.py:1070


        How can I get the undercloud back to a clean state?


        __________________________________________________________________________

        OpenStack Development Mailing List (not for usage questions)
        Unsubscribe:
        OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
        
        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




    __________________________________________________________________________

    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
    
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 29, 2016 by Adam_Young (19,940 points)   2 7 9
...